Multivariate Statistics
Summary of all formulas, proofs, recipes and notation
Levi S
Master of Science
Based on Johnson & Wichern (6th ed.), the lectures (Zhelonkin, EUR),
the tutorials and the IMS summary
How to use this document
• Finding things: clickable table of contents, Recipes (orange, R1–R15, overview in §14), formula
sheet (§13), tables (§17), a Dutch–English glossary and the index at the back.
• Colours: blue = definition, green = theorem/result, grey = proof, red = pitfall or error in a source,
orange = step-by-step recipe, purple = how to write it down, teal = test yourself.
• Learning, not copying: tutorial exercises are not solved here. §15 lists, per exercise, which theory
and recipe you need, plus a hint. Theory from the book and lectures is fully proved.
• Critical reading: the sources contain errors (including the IMS summary and some slides). These
are marked in red boxes and collected in §16. When in doubt, J&W is leading.
Notation follows J&W and the lectures; ⊤ = transpose (J&W writes ′ ).
,Levi S, M.Sc. Contents
Contents
1 Notation conventions (read this first) 3
2 Linear algebra and matrix toolkit 3
2.1 Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Matrix multiplication by hand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 Partitioned matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.4 Eigenvalues and spectral decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.5 Maximising quadratic forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.6 Matrix derivatives (lecture convention) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3 Distance and statistical distance (week 1) 7
3.1 Statistical distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4 Random vectors (week 1) 8
4.1 Linear combinations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
5 Sample geometry, estimating µ and Σ (weeks 1–2) 9
5.1 Sample statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.2 Geometry: rows and columns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.3 Unbiasedness of x̄ and S . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5.4 Generalised variance (week 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6 The multivariate normal distribution (weeks 2–3) 12
6.1 Properties (J&W Result 4.1–4.7) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
6.2 Conditional distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
6.3 Quadratic form and probability contours . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
6.4 Step-by-step: drawing an ellipse (contour or confidence region) . . . . . . . . . . . . . . . . . . 15
6.5 Other useful facts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
7 Maximum likelihood and sampling distributions (week 3) 17
7.1 General principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
7.2 The log-likelihood of the multivariate normal . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
7.3 Sampling distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
7.4 Large-sample results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
8 Checking normality (week 3) 20
8.1 Univariate: Q-Q plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
8.2 Multivariate: chi-square plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
8.3 Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
9 Tests for µ: Hotelling’s T 2 and the LRT (week 4) 21
9.1 Univariate (recap) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
9.2 Hotelling’s T2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
9.3 Likelihood ratio test (LRT) and Wilks’ lambda . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Multivariate Statistics — summary (open book) 1
,Levi S, M.Sc. Contents
10 Confidence regions and simultaneous intervals (weeks 4–5) 23
10.1 Simultaneous T2 intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
10.2 Bonferroni . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
10.3 Large samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
10.4 Comparison and drawing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
11 Principal component analysis (PCA) 26
11.1 Standardised variables and scale dependence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
11.2 Special structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
11.3 Sample PCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
11.4 How many components? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
12 Factor analysis (FA) 28
12.1 Estimation: principal component method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
12.2 Factor scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
13 Formula sheet (at a glance) 30
14 Overview of all recipes 31
15 Tutorial guide (hints, no solutions) 31
16 Errata: errors and ambiguities in the sources 32
17 Tables 33
17.1 χ2ν (α) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
( n −1) p
17.2 c2 = n− p Fp,n− p (α) (critical value T 2 , confidence region constant) . . . . . . . . . . . . . . . . 34
17.3 Fν1 ,ν2 (α) for α = 0.10, 0.05, 0.01 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
17.4 tν (α) (one-sided upper percentiles) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
17.5 Standard normal quantiles Φ −1 ( q ) (for Q-Q plots) . . . . . . . . . . . . . . . . . . . . . . . . . . 37
18 Glossary Dutch–English (exams are bilingual) 37
Index 38
Multivariate Statistics — summary (open book) 2
, Levi S, M.Sc. 2 Linear algebra and matrix toolkit
1 Notation conventions (read this first)
Notation & write-up: general
• Vectors are column vectors: x = ( x1 , . . . , x p )⊤ ∈ R p×1 . By hand: underline vectors (x) and matrices
(A), or state the dimension. Always state the dimensions of a new object, e.g. C ∈ R2×3 .
• Transpose: A⊤ (lectures, exams) = A′ (J&W). Pick one and be consistent.
• Data matrix X ∈ Rn× p : row j = observation x⊤
j , column k = variable k. Element x jk = observation
j, variable k.
• Vector of ones: 1n = (1, . . . , 1)⊤ . The lectures write ι, the IMS summary u, J&W 1. All the same
thing.
• Population vs. sample: µ, Σ, ρik , λi , ei (population) vs. x̄, S, rik , λ̂i , êi (sample).
• Three covariance estimators — watch the divisor:
n −1
S= 1
n −1 ∑ j ( x j − x̄)( x j − x̄)⊤ (unbiased), Sn = Σ̂MLE = 1
n ∑ j ( x j − x̄)( x j − x̄)⊤ = n S.
Exercises sometimes write Σ̂ for S and sometimes for Sn . Always check which divisor is used and
say in your answer which one you use.
• Critical values (upper percentiles): Fp,ν (α), χ2p (α), tν (α) are the values with probability α to the
right. So P(χ2p > χ2p (α)) = α. In R: qf(1-a,p,nu), qchisq(1-a,p), qt(1-a,nu).
• Distributions: x ∼ N p (µ, Σ): the subscript is the dimension. The second argument is the covariance
matrix (variance, not standard deviation).
• Determinant | A| = det A; trace tr A = ∑i aii .
Notation & write-up: a complete exam answer
On the exam: answers without motivation are marked wrong. So for every calculation write down:
iid
1. Assumption/model (e.g. “x1 , . . . , xn ∼ N p (µ, Σ)”).
2. Which result you use, by name or formula (e.g. “Result 4.3: Cx ∼ Nq (Cµ, CΣC ⊤ )”).
3. Substitution with numbers, including intermediate results (matrices written out).
4. Critical value with degrees of freedom and α, e.g. F2,8 (0.05) = 4.46.
5. Decision + conclusion in words (“we do not reject H0 at the 5% level; there is insufficient evidence
that. . . ”).
When drawing (ellipse/intervals): centre, directions (ei ), half-axis lengths with formula, and label the
axes.
2 Linear algebra and matrix toolkit
2.1 Rules
• ( AB)⊤ = B⊤ A⊤ , ( AB)−1 = B−1 A−1 , ( A ⊤ ) −1 = ( A −1 ) ⊤ .
• | AB| = | A|| B|, | A ⊤ | = | A |, | A −1 | = | A | −1 , |cA| = c p | A| for A ∈ R p× p .
• tr( AB) = tr( BA) (cyclic), tr( A + B) = tr A + tr B, tr(c) = c for a scalar.
• Trace trick: a quadratic form is a scalar, so x⊤ Ax = tr( x⊤ Ax) = tr( Axx⊤ ).
• rank( AB) ≤ min{rank A, rank B}.
• Determinant lemma (Sylvester): for A ∈ R p×q , B ∈ Rq× p : | I p + AB| = | Iq + BA|. In particular:
| I p + ab⊤ | = 1 + b⊤ a and | B + dd⊤ | = | B| (1 + d⊤ B−1 d).
−1
a b 1 d −b
• Inverse 2 × 2: = .
c d ad − bc −c a
Multivariate Statistics — summary (open book) 3
Summary of all formulas, proofs, recipes and notation
Levi S
Master of Science
Based on Johnson & Wichern (6th ed.), the lectures (Zhelonkin, EUR),
the tutorials and the IMS summary
How to use this document
• Finding things: clickable table of contents, Recipes (orange, R1–R15, overview in §14), formula
sheet (§13), tables (§17), a Dutch–English glossary and the index at the back.
• Colours: blue = definition, green = theorem/result, grey = proof, red = pitfall or error in a source,
orange = step-by-step recipe, purple = how to write it down, teal = test yourself.
• Learning, not copying: tutorial exercises are not solved here. §15 lists, per exercise, which theory
and recipe you need, plus a hint. Theory from the book and lectures is fully proved.
• Critical reading: the sources contain errors (including the IMS summary and some slides). These
are marked in red boxes and collected in §16. When in doubt, J&W is leading.
Notation follows J&W and the lectures; ⊤ = transpose (J&W writes ′ ).
,Levi S, M.Sc. Contents
Contents
1 Notation conventions (read this first) 3
2 Linear algebra and matrix toolkit 3
2.1 Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Matrix multiplication by hand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 Partitioned matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.4 Eigenvalues and spectral decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.5 Maximising quadratic forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.6 Matrix derivatives (lecture convention) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3 Distance and statistical distance (week 1) 7
3.1 Statistical distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4 Random vectors (week 1) 8
4.1 Linear combinations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
5 Sample geometry, estimating µ and Σ (weeks 1–2) 9
5.1 Sample statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.2 Geometry: rows and columns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.3 Unbiasedness of x̄ and S . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5.4 Generalised variance (week 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6 The multivariate normal distribution (weeks 2–3) 12
6.1 Properties (J&W Result 4.1–4.7) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
6.2 Conditional distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
6.3 Quadratic form and probability contours . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
6.4 Step-by-step: drawing an ellipse (contour or confidence region) . . . . . . . . . . . . . . . . . . 15
6.5 Other useful facts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
7 Maximum likelihood and sampling distributions (week 3) 17
7.1 General principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
7.2 The log-likelihood of the multivariate normal . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
7.3 Sampling distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
7.4 Large-sample results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
8 Checking normality (week 3) 20
8.1 Univariate: Q-Q plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
8.2 Multivariate: chi-square plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
8.3 Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
9 Tests for µ: Hotelling’s T 2 and the LRT (week 4) 21
9.1 Univariate (recap) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
9.2 Hotelling’s T2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
9.3 Likelihood ratio test (LRT) and Wilks’ lambda . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Multivariate Statistics — summary (open book) 1
,Levi S, M.Sc. Contents
10 Confidence regions and simultaneous intervals (weeks 4–5) 23
10.1 Simultaneous T2 intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
10.2 Bonferroni . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
10.3 Large samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
10.4 Comparison and drawing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
11 Principal component analysis (PCA) 26
11.1 Standardised variables and scale dependence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
11.2 Special structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
11.3 Sample PCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
11.4 How many components? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
12 Factor analysis (FA) 28
12.1 Estimation: principal component method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
12.2 Factor scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
13 Formula sheet (at a glance) 30
14 Overview of all recipes 31
15 Tutorial guide (hints, no solutions) 31
16 Errata: errors and ambiguities in the sources 32
17 Tables 33
17.1 χ2ν (α) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
( n −1) p
17.2 c2 = n− p Fp,n− p (α) (critical value T 2 , confidence region constant) . . . . . . . . . . . . . . . . 34
17.3 Fν1 ,ν2 (α) for α = 0.10, 0.05, 0.01 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
17.4 tν (α) (one-sided upper percentiles) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
17.5 Standard normal quantiles Φ −1 ( q ) (for Q-Q plots) . . . . . . . . . . . . . . . . . . . . . . . . . . 37
18 Glossary Dutch–English (exams are bilingual) 37
Index 38
Multivariate Statistics — summary (open book) 2
, Levi S, M.Sc. 2 Linear algebra and matrix toolkit
1 Notation conventions (read this first)
Notation & write-up: general
• Vectors are column vectors: x = ( x1 , . . . , x p )⊤ ∈ R p×1 . By hand: underline vectors (x) and matrices
(A), or state the dimension. Always state the dimensions of a new object, e.g. C ∈ R2×3 .
• Transpose: A⊤ (lectures, exams) = A′ (J&W). Pick one and be consistent.
• Data matrix X ∈ Rn× p : row j = observation x⊤
j , column k = variable k. Element x jk = observation
j, variable k.
• Vector of ones: 1n = (1, . . . , 1)⊤ . The lectures write ι, the IMS summary u, J&W 1. All the same
thing.
• Population vs. sample: µ, Σ, ρik , λi , ei (population) vs. x̄, S, rik , λ̂i , êi (sample).
• Three covariance estimators — watch the divisor:
n −1
S= 1
n −1 ∑ j ( x j − x̄)( x j − x̄)⊤ (unbiased), Sn = Σ̂MLE = 1
n ∑ j ( x j − x̄)( x j − x̄)⊤ = n S.
Exercises sometimes write Σ̂ for S and sometimes for Sn . Always check which divisor is used and
say in your answer which one you use.
• Critical values (upper percentiles): Fp,ν (α), χ2p (α), tν (α) are the values with probability α to the
right. So P(χ2p > χ2p (α)) = α. In R: qf(1-a,p,nu), qchisq(1-a,p), qt(1-a,nu).
• Distributions: x ∼ N p (µ, Σ): the subscript is the dimension. The second argument is the covariance
matrix (variance, not standard deviation).
• Determinant | A| = det A; trace tr A = ∑i aii .
Notation & write-up: a complete exam answer
On the exam: answers without motivation are marked wrong. So for every calculation write down:
iid
1. Assumption/model (e.g. “x1 , . . . , xn ∼ N p (µ, Σ)”).
2. Which result you use, by name or formula (e.g. “Result 4.3: Cx ∼ Nq (Cµ, CΣC ⊤ )”).
3. Substitution with numbers, including intermediate results (matrices written out).
4. Critical value with degrees of freedom and α, e.g. F2,8 (0.05) = 4.46.
5. Decision + conclusion in words (“we do not reject H0 at the 5% level; there is insufficient evidence
that. . . ”).
When drawing (ellipse/intervals): centre, directions (ei ), half-axis lengths with formula, and label the
axes.
2 Linear algebra and matrix toolkit
2.1 Rules
• ( AB)⊤ = B⊤ A⊤ , ( AB)−1 = B−1 A−1 , ( A ⊤ ) −1 = ( A −1 ) ⊤ .
• | AB| = | A|| B|, | A ⊤ | = | A |, | A −1 | = | A | −1 , |cA| = c p | A| for A ∈ R p× p .
• tr( AB) = tr( BA) (cyclic), tr( A + B) = tr A + tr B, tr(c) = c for a scalar.
• Trace trick: a quadratic form is a scalar, so x⊤ Ax = tr( x⊤ Ax) = tr( Axx⊤ ).
• rank( AB) ≤ min{rank A, rank B}.
• Determinant lemma (Sylvester): for A ∈ R p×q , B ∈ Rq× p : | I p + AB| = | Iq + BA|. In particular:
| I p + ab⊤ | = 1 + b⊤ a and | B + dd⊤ | = | B| (1 + d⊤ B−1 d).
−1
a b 1 d −b
• Inverse 2 × 2: = .
c d ad − bc −c a
Multivariate Statistics — summary (open book) 3
