Complete Course Summary
OLS in Simple and Matrix Form, Gauss–Markov, Inference, Asymptotics,
Heteroskedasticity, Instrumental Variables & GMM, Maximum Likelihood,
Binary, Multinomial and Ordered Choice, Truncated and Censored Data
Lecture Notes, Proofs, Step-by-Step Exam Guides
& Worked Practice Exercises with Full Solutions
How to use this document. Part I is the theory in the order of the course, with
the key derivations written out. Part II compresses everything into checklists and
decision tables for the exam. Part III contains original practice exercises with full
solutions, ranging from computational questions to proof questions. Boxes mark key
results, exam tips and common pitfalls.
,Econometrics 1 Contents
Contents
I Lecture Notes & Derivations 4
1 Simple Linear Regression and OLS 4
1.1 Set-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Deriving the OLS estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Linear transformations of the data . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Treatment effect (dummy regressor) . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.5 Goodness of fit: R2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.6 Why include a constant? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Gauss–Markov Assumptions (Finite Sample) 5
2.1 Bias and variance in the simple model . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Unbiased estimation of σ02 (simple model) . . . . . . . . . . . . . . . . . . . . . . . 6
3 Multiple Regression in Matrix Form 7
3.1 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.2 The OLS estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.3 Geometry: projections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.4 Gauss–Markov assumptions in matrix form . . . . . . . . . . . . . . . . . . . . . . 8
3.5 Bias and variance of OLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.6 Gauss–Markov theorem: OLS is BLUE . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.7 Variance estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.8 R2 in matrix form and adjusted R2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4 Frisch–Waugh–Lovell, Omitted Variables, Multicollinearity 9
4.1 Frisch–Waugh–Lovell (FWL) theorem . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.2 Restricted OLS and omitted variable bias . . . . . . . . . . . . . . . . . . . . . . . 9
4.3 Multicollinearity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5 OLS under Normality and Exact Inference 10
5.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5.2 Distributions under A7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5.3 Testing a single coefficient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.4 Joint linear restrictions: the F -test . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
6 Asymptotic Theory 11
6.1 Convergence concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
6.2 OLS is consistent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
6.3 OLS is asymptotically normal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
6.4 Consequences for t-statistics and the Wald test . . . . . . . . . . . . . . . . . . . . 13
1
,Econometrics 1 Contents
7 Heteroskedasticity 13
7.1 What still holds, and what does not . . . . . . . . . . . . . . . . . . . . . . . . . . 13
7.2 White heteroskedasticity-robust standard errors . . . . . . . . . . . . . . . . . . . . 14
7.3 Generalised, weighted and feasible weighted least squares . . . . . . . . . . . . . . 14
7.4 Tests for heteroskedasticity and non-normality . . . . . . . . . . . . . . . . . . . . 14
8 Model Specification and Interpretation 15
8.1 Model-selection procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
8.2 Functional forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
8.3 Worked interpretation: Mincer equation and interactions . . . . . . . . . . . . . . . 15
8.4 Hypothesis-test recipe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
9 Endogeneity, Instrumental Variables and GMM 16
9.1 The problem and the instruments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
9.2 GMM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
9.3 Tests in the IV framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
10 Maximum Likelihood 17
11 Binary Choice Models 18
12 Multinomial and Ordered Choice 18
12.1 Unordered outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
12.2 Ordered outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
13 Truncated and Censored Data 19
13.1 Truncated data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
13.2 Censored data: the Tobit model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
II Quick-Reference Step-by-Step Guides 21
14 Guide: Which Test Do I Use? 21
15 Guide: Carrying Out a Hypothesis Test (Exam Format) 21
16 Guide: OLS Proof Toolbox 22
17 Guide: Choosing and Interpreting Functional Forms 22
18 Guide: Heteroskedasticity Workflow 23
19 Guide: IV / 2SLS / GMM Workflow 23
20 Guide: ML, Discrete-Choice and Censoring Workflow 23
2
,Econometrics 1 Contents
21 Formula Sheet 24
22 Common Mistakes Examiners Look For 24
III Practice Exercises with Full Solutions 26
23 How to Use the Exercises 26
24 Block A: OLS, Finite-Sample Theory and Exact Inference 26
25 Block B: Asymptotics and Heteroskedasticity 29
26 Block C: Instrumental Variables and GMM 30
27 Block D: ML, Discrete Choice, Truncation and Censoring 31
28 Mixed Review: Short Questions 34
3
,Econometrics 1 1 Simple Linear Regression and OLS
Part I
Lecture Notes & Derivations
1 Simple Linear Regression and OLS
1.1 Set-up
Population
P parameters are written α, β; their estimates α̂, β̂ (or a, b). Inner product: a′ b = b′ a =
i ai bi .
Model. For i = 1, . . . , n (i.i.d. sample):
yi = α + βxi + εi ,
with yi the dependent (target) variable, xi the regressor, n the sample size and εi the error term:
the variation
P in yi that cannot be linearly explained by the observed regressors. With K regressors:
yi = α + K k=1 βk xk,i + εi .
P
Marginal effect. If E[εi | xi ] = 0 then E[yi | x1,i , . . . , xK,i ] = α + k βk xk,i and hence
∂E[yi | x1,i , . . . , xK,i ]
= βk ,
∂xk,i
i.e. βk is the effect of xk on y given all other regressors. For OLS to identify a causal relation the
true relationship must be linear (proportional changes).
Interpolation = predicting within the range of the data used to fit the model; extrapolation =
predicting outside that range (risky).
1.2 Deriving the OLS estimator
Minimise the sum of squared residuals Sn (α, β) = i (yi − α − βxi )2 , with ε̂i = êi = yi − ŷi =
P
yi − α̂ − β̂xi .
P P
FOC w.r.t. α: −2 i (yi − α − βxi ) = 0 ⇒ nα = i (yi − βxi ) ⇒ α̂ = ȳ − β̂ x̄.
P
FOC w.r.t. β: −2 i xi (yi − α − βxi ) = 0. Substitute α̂ = ȳ − β̂ x̄:
P P
X
i xi (yi − ȳ) (xi − x̄)(yi − ȳ) Sxy
= iP
xi yi − ȳ − β̂(xi − x̄) = 0 ⇒ β̂ = P 2
= 2,
i i xi (xi − x̄) i (xi − x̄) Sx
1 1
(xi − x̄)(yi − ȳ) and Sx2 = (xi − x̄)2 . (Use (xi − x̄) = 0 to swap xi for
P P P
where Sxy = n−1 n−1
xi − x̄.)
Need to know (simple regression)
P
(xi − x̄)(yi − ȳ) Sxy
β̂ = P 2
= 2 , α̂ = ȳ − β̂ x̄, ŷi = α̂ + β̂xi , êi = yi − ŷi .
(xi − x̄) Sx
X X X X X
êi = 0, xi êi = 0, ŷi êi = 0, yi = ŷi .
i i i i i
P
(The condition i ŷi êi = 0 says fitted values and residuals are uncorrelated.)
P P P P
Orthogonality (geometry). ŷi êi = (α̂ + β̂xi )êi = α̂ êi + β̂ xi êi = 0 because both
normal equations equal zero.
4
,Econometrics 1 2 Gauss–Markov Assumptions (Finite Sample)
1.3 Linear transformations of the data
Let yi∗ = ay + by yi and x∗i = ax + bx xi and regress y ∗ on x∗ :
by by ax
β̂ ∗ = β̂, α̂∗ = ay + by α̂ − β̂, ê∗i = by êi .
bx bx
Summary
α̂ is affected by both scale and location; β̂ is affected by scale only. Demeaning both variables
(ỹi = yi − ȳ, x̃i = xi − x̄) and regressing without a constant gives the same β̂ and the same
residuals.
Proof of β̂ ∗ . Plug in: x∗i − x̄∗ = bx (xi − x̄) and yi∗ − ȳ ∗ = by (yi − ȳ); the ratio gives by bx /b2x = by /bx .
1.4 Treatment effect (dummy regressor)
yi = α + Di β + εi , Di = 1 if treated, 0 otherwise; n = n0 + n1 . Under random assignment OLS
gives
β̂ = ȳ1 − ȳ0 (difference of the two sample means), α̂ = ȳ0 .
P
(Use D̄ = n1 /n, Di yi = n1 ȳ1 , ȳ = (n0 ȳ0 + n1 ȳ1 )/n.)
1.5 Goodness of fit: R2
P
Since yi = ŷi + êi and êi = 0,
X X X
(yi − ȳ)2 = (ŷi − ȳ)2 + ê2i
P
(cross term 2 êi (ŷi − ȳ) = 0).
| {z } | {z } | {z }
SST SSE (explained) SSR (unexplained)
SSE SSR
R2 = =1− ∈ [0, 1].
SST SST
OLS minimises SSR ⇔ maximises SSE ⇔ maximises R2 .
P 2 2
Sxy
[ (x i −x̄)(yi −ȳ)]
2 . SSE = β̂ 2
Simple regression: R2 = rxy (xi − x̄)2 = so R2 = 2 .
P
P
(xi −x̄)2
, Sx2 Sy2
= rxy
Pitfall: choosing a model by R2
A high R2 says nothing about out-of-sample prediction, so it does not guarantee better pre-
dictions. To choose a model one must consider in-sample fit, out-of-sample performance and
non-statistical considerations.
1.6 Why include a constant?
Dropping α has drawbacks: (i) OLS is no longer related to the correlation coefficient; (ii) OLS is
not invariant to transformations; (iii) if the true constant is non-zero, β̂ is biased and inconsistent;
(iv) R2 is no longer the squared correlation; (v) with a binary regressor you lose a reference group.
The only upside is lower variance when the true intercept is zero, which is hardly ever the case.
2 Gauss–Markov Assumptions (Finite Sample)
To answer questions on bias, consistency and variance, treat α̂, β̂ as random variables.
5