Linear Models in Statistics
Theory
EBB072A05
Semester I B
Wouter Voskuilen
S4916344
Slides by dr. M. Kesina
1
,Wouter Voskuilen Linear Models in Statistics
Contents
1 Chapter 1: Matrix Algebra 3
1.1 Vectors and Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Rank, Determinant, and Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 Positive (semi) Definite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.5 Projection Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.6 Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.7 Differentiation Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2 Chapter 2: Random Vectors 17
2.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Operations and Transformations . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Expected Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4 Variance-covariance matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.5 Correlation matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.6 General Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3 Chapter 3: Multivariate Normal and Related Distributions 23
3.1 Univariate Normal Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.2 Multivariate Normal Distribution . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3 Related Distributions: χ2 , F , and t distribution . . . . . . . . . . . . . . . . . 25
3.4 Results on Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.5 Distribution Check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4 Chapter 4: Linear Model 28
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.2 OLS Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.2.1 Bivariate Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.2.2 Multivariate Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.3 Goodness of Fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.4 Properties of the OLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.5 Hypothesis Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.6 Maximum Likelihood Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.7 Frisch-Waugh-Lovell Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.8 Wrong Specification of the Regressor Matrix . . . . . . . . . . . . . . . . . . . 56
4.9 Violation of the OLS Assumptions . . . . . . . . . . . . . . . . . . . . . . . . 60
4.9.1 Violation of Assumption A.2 . . . . . . . . . . . . . . . . . . . . . . . 60
4.9.2 Violation of Assumption A.4 . . . . . . . . . . . . . . . . . . . . . . . 60
2
,Wouter Voskuilen Linear Models in Statistics
1 Chapter 1: Matrix Algebra
1.1 Vectors and Matrices
Let
a1
a2
a= .
..
an
a is called a vector (column vector).
The elements ai for i = 1, ..., n are called elements or components of a. n is the order of the
vector.
Let a and b be two vectors, which have the same order n.
Summation of two vectors:
a1 b1 a1 + b1
a2 b2 a2 + b2
a+b= . + . = .
.. .. ..
an bn an + bn
Vector summation is:
− Commutative: a + b = b + a
− Associative: (a + b) + c = a + (b + c), where c is a vector of the same order as a and
b.
Multiplication of a vector with a scalar λ
a1
a2
λa = λ .
..
an
Inner or scalar product of two vectors a and b of the same order n
n
X
⟨a, b⟩ = a′ b = ai bi .
i=1
The length (or norm) of a vector
√
∥a∥ = ⟨a, a⟩1/2 = a′ a.
3
, Wouter Voskuilen Linear Models in Statistics
Any nonzero vector can be normalized by
1
ao = a.
∥a∥
A normalized vector has norm 1.
Collinearity of two vectors a and b
a = λb
for some scalar λ.
Two vectors a and b with ⟨a, b⟩ = 0 are called orthogonal.
If ⟨a, b⟩ = 0 and ∥a∥ = ∥b∥ = 1, then a and b are called orthonormal.
Outer product of two vectors a and b of the same order
a1 b1 · · · a1 bn
ab′ = ... .. ..
. .
an b1 · · · an bn
Unit vectors or Elementary vectors ej :
ej consists of zeros and a single one on the jth position.
Unit vectors are orthonormal.
Vector of ones ιn :
ιn consists of ones only. The index indicates the size.
ιn is also called the sum vector
n
X n
X
ι′n a = ιi a i = ai .
i=1 i=1
Let
a11 · · · a1m
.. .. ..
A= . . .
a1n · · · anm
The matrix A is of order/size/dimension n × m.
We also write A = {aij } and Aij = aij .
Let A be a n × m matrix.
− A is square if n = m.
− A is symmetric if n = m and aij = aji , i, j = 1, ..., n.
4
Theory
EBB072A05
Semester I B
Wouter Voskuilen
S4916344
Slides by dr. M. Kesina
1
,Wouter Voskuilen Linear Models in Statistics
Contents
1 Chapter 1: Matrix Algebra 3
1.1 Vectors and Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Rank, Determinant, and Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 Positive (semi) Definite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.5 Projection Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.6 Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.7 Differentiation Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2 Chapter 2: Random Vectors 17
2.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Operations and Transformations . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Expected Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4 Variance-covariance matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.5 Correlation matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.6 General Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3 Chapter 3: Multivariate Normal and Related Distributions 23
3.1 Univariate Normal Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.2 Multivariate Normal Distribution . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3 Related Distributions: χ2 , F , and t distribution . . . . . . . . . . . . . . . . . 25
3.4 Results on Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.5 Distribution Check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4 Chapter 4: Linear Model 28
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.2 OLS Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.2.1 Bivariate Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.2.2 Multivariate Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.3 Goodness of Fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.4 Properties of the OLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.5 Hypothesis Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.6 Maximum Likelihood Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.7 Frisch-Waugh-Lovell Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.8 Wrong Specification of the Regressor Matrix . . . . . . . . . . . . . . . . . . . 56
4.9 Violation of the OLS Assumptions . . . . . . . . . . . . . . . . . . . . . . . . 60
4.9.1 Violation of Assumption A.2 . . . . . . . . . . . . . . . . . . . . . . . 60
4.9.2 Violation of Assumption A.4 . . . . . . . . . . . . . . . . . . . . . . . 60
2
,Wouter Voskuilen Linear Models in Statistics
1 Chapter 1: Matrix Algebra
1.1 Vectors and Matrices
Let
a1
a2
a= .
..
an
a is called a vector (column vector).
The elements ai for i = 1, ..., n are called elements or components of a. n is the order of the
vector.
Let a and b be two vectors, which have the same order n.
Summation of two vectors:
a1 b1 a1 + b1
a2 b2 a2 + b2
a+b= . + . = .
.. .. ..
an bn an + bn
Vector summation is:
− Commutative: a + b = b + a
− Associative: (a + b) + c = a + (b + c), where c is a vector of the same order as a and
b.
Multiplication of a vector with a scalar λ
a1
a2
λa = λ .
..
an
Inner or scalar product of two vectors a and b of the same order n
n
X
⟨a, b⟩ = a′ b = ai bi .
i=1
The length (or norm) of a vector
√
∥a∥ = ⟨a, a⟩1/2 = a′ a.
3
, Wouter Voskuilen Linear Models in Statistics
Any nonzero vector can be normalized by
1
ao = a.
∥a∥
A normalized vector has norm 1.
Collinearity of two vectors a and b
a = λb
for some scalar λ.
Two vectors a and b with ⟨a, b⟩ = 0 are called orthogonal.
If ⟨a, b⟩ = 0 and ∥a∥ = ∥b∥ = 1, then a and b are called orthonormal.
Outer product of two vectors a and b of the same order
a1 b1 · · · a1 bn
ab′ = ... .. ..
. .
an b1 · · · an bn
Unit vectors or Elementary vectors ej :
ej consists of zeros and a single one on the jth position.
Unit vectors are orthonormal.
Vector of ones ιn :
ιn consists of ones only. The index indicates the size.
ιn is also called the sum vector
n
X n
X
ι′n a = ιi a i = ai .
i=1 i=1
Let
a11 · · · a1m
.. .. ..
A= . . .
a1n · · · anm
The matrix A is of order/size/dimension n × m.
We also write A = {aij } and Aij = aij .
Let A be a n × m matrix.
− A is square if n = m.
− A is symmetric if n = m and aij = aji , i, j = 1, ..., n.
4
