Summary Linear Models in Statistics | All Theory from Slides

Linear Models in Statistics
Theory
EBB072A05
Semester I B


Wouter Voskuilen
S4916344
Slides by dr. M. Kesina




1

,Wouter Voskuilen Linear Models in Statistics


Contents
1 Chapter 1: Matrix Algebra 3
1.1 Vectors and Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Rank, Determinant, and Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 Positive (semi) Definite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.5 Projection Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.6 Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.7 Differentiation Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2 Chapter 2: Random Vectors 17
2.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Operations and Transformations . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Expected Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4 Variance-covariance matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.5 Correlation matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.6 General Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3 Chapter 3: Multivariate Normal and Related Distributions 23
3.1 Univariate Normal Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.2 Multivariate Normal Distribution . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3 Related Distributions: χ2 , F , and t distribution . . . . . . . . . . . . . . . . . 25
3.4 Results on Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.5 Distribution Check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4 Chapter 4: Linear Model 28
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.2 OLS Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.2.1 Bivariate Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.2.2 Multivariate Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.3 Goodness of Fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.4 Properties of the OLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.5 Hypothesis Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.6 Maximum Likelihood Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.7 Frisch-Waugh-Lovell Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.8 Wrong Specification of the Regressor Matrix . . . . . . . . . . . . . . . . . . . 56
4.9 Violation of the OLS Assumptions . . . . . . . . . . . . . . . . . . . . . . . . 60
4.9.1 Violation of Assumption A.2 . . . . . . . . . . . . . . . . . . . . . . . 60
4.9.2 Violation of Assumption A.4 . . . . . . . . . . . . . . . . . . . . . . . 60




2

,Wouter Voskuilen Linear Models in Statistics


1 Chapter 1: Matrix Algebra
1.1 Vectors and Matrices
Let  
a1
 a2 
a= . 
 
 .. 
an
a is called a vector (column vector).
The elements ai for i = 1, ..., n are called elements or components of a. n is the order of the
vector.

Let a and b be two vectors, which have the same order n.
Summation of two vectors:
     
a1 b1 a1 + b1
 a2   b2   a2 + b2 
a+b= . + . = . 
     
 ..   ..   .. 
an bn an + bn

Vector summation is:

− Commutative: a + b = b + a

− Associative: (a + b) + c = a + (b + c), where c is a vector of the same order as a and
b.

Multiplication of a vector with a scalar λ
 
a1
 a2 
λa = λ  . 
 
 .. 
an

Inner or scalar product of two vectors a and b of the same order n
n
X
⟨a, b⟩ = a′ b = ai bi .
i=1

The length (or norm) of a vector
√
∥a∥ = ⟨a, a⟩1/2 = a′ a.


3

, Wouter Voskuilen Linear Models in Statistics


Any nonzero vector can be normalized by
1
ao = a.
∥a∥
A normalized vector has norm 1.

Collinearity of two vectors a and b
a = λb
for some scalar λ.

Two vectors a and b with ⟨a, b⟩ = 0 are called orthogonal.
If ⟨a, b⟩ = 0 and ∥a∥ = ∥b∥ = 1, then a and b are called orthonormal.

Outer product of two vectors a and b of the same order
 
a1 b1 · · · a1 bn
ab′ =  ... .. .. 

. . 
an b1 · · · an bn

Unit vectors or Elementary vectors ej :
ej consists of zeros and a single one on the jth position.
Unit vectors are orthonormal.

Vector of ones ιn :
ιn consists of ones only. The index indicates the size.
ιn is also called the sum vector
n
X n
X
ι′n a = ιi a i = ai .
i=1 i=1


Let  
a11 · · · a1m
 .. .. .. 
A= . . . 
a1n · · · anm
The matrix A is of order/size/dimension n × m.
We also write A = {aij } and Aij = aij .

Let A be a n × m matrix.
− A is square if n = m.

− A is symmetric if n = m and aij = aji , i, j = 1, ..., n.

4