CS229 Machine Learning — Study Guide 1
CS229 Machine Learning
Study Guide
Intermediate Level • Stanford CS229 Core Topics
This guide consolidates the four core pillars of Stanford’s CS229: supervised learning, learning
theory, unsupervised learning, and neural networks. It assumes comfort with linear algebra
(matrix operations, eigendecomposition), multivariable calculus (gradients, Hessians), and basic
probability (Gaussians, Bayes’ rule). Each section ends with a practice problem; complete
solutions are provided in the final section so you can attempt each problem before checking your
work. Read actively — re-derive every gradient and re-sketch every model.
Contents
1 Linear Regression & Least Squares 2
2 Logistic Regression & Generalized Linear Models 3
3 Generative Models: GDA & Naive Bayes 4
4 Support Vector Machines & Kernels 4
5 Learning Theory 5
6 Unsupervised Learning 6
7 Neural Networks & Backpropagation 7
8 Solutions to Practice Problems 9
9 Summary & Further Reading 11
, CS229 Machine Learning — Study Guide 2
1. Linear Regression & Least Squares
Linear regression is the foundation of supervised learning: predict a real-valued output y ∈ R
as a linear function of features x ∈ Rn. The hypothesis is
n
Σ
hθ(x) = θ ⊤x = θjxj,
j=0
where we append a bias feature x0 = 1. Given a training set {(x(i), y(i))}mi=1 , we choose θ to
minimize the squared-error cost
m
1Σ 2
J(θ) = hθ (x(i)) − y(i) .
2
i=1
Gradient descent updates θj := θj − α ∂J∂θ,j where for a single example
∂J
= h θ(x) − y xj .
∂θj
This is the LMS (least-mean-squares) update. The update is proportional to the error (hθ(x)−y),
which gives the algorithm its self-correcting character.
Batch gradient descent sums the gradient over all m examples; stochastic gradient descent
(SGD) updates after each example, which is cheaper and often faster on large datasets. The
learning rate α controls step size — too small and convergence is slow; too large and the cost
may diverge.
The normal equations give the closed-form minimizer. Stacking the data into a design matrix
⊤
X ∈ Rm×(n+1) (rows are x(i) ) and label vector y ∈ Rm, the optimal θ satisfies
∇θJ = X⊤(Xθ − y) = 0 =⇒ θ = (X⊤X)−1X⊤y.
This is the maximum-likelihood estimator under Gaussian noise y(i) | x(i) ∼ N (θ⊤x(i), σ2).
Key formula: normal equations
Closed-form least-squares solution. Requires X⊤X to be invertible (features linearly inde-
Worked mini-example. Two points: (x, y) = (1, 2), (2, 4), with bias feature. Then X =
1 1 2 3 5 −3
, y = 24 . Compute X⊤X = , X⊤y = 610 . Inverse of X⊤X is (det
1 2 3 5 −3 2
5·6−3·10
= 1), giving θ = −3·6+2·10 = 20 . So the line is h(x) = 0 + 2x = 2x, exactly fitting both points.
Practice Problem 1
{(1, 3), (2, 5), (3, 7)}. Compute the least-squares parameter vector θ (θ0, θ1) using the
normal equations. Verify that the residuals are zero.
CS229 Machine Learning
Study Guide
Intermediate Level • Stanford CS229 Core Topics
This guide consolidates the four core pillars of Stanford’s CS229: supervised learning, learning
theory, unsupervised learning, and neural networks. It assumes comfort with linear algebra
(matrix operations, eigendecomposition), multivariable calculus (gradients, Hessians), and basic
probability (Gaussians, Bayes’ rule). Each section ends with a practice problem; complete
solutions are provided in the final section so you can attempt each problem before checking your
work. Read actively — re-derive every gradient and re-sketch every model.
Contents
1 Linear Regression & Least Squares 2
2 Logistic Regression & Generalized Linear Models 3
3 Generative Models: GDA & Naive Bayes 4
4 Support Vector Machines & Kernels 4
5 Learning Theory 5
6 Unsupervised Learning 6
7 Neural Networks & Backpropagation 7
8 Solutions to Practice Problems 9
9 Summary & Further Reading 11
, CS229 Machine Learning — Study Guide 2
1. Linear Regression & Least Squares
Linear regression is the foundation of supervised learning: predict a real-valued output y ∈ R
as a linear function of features x ∈ Rn. The hypothesis is
n
Σ
hθ(x) = θ ⊤x = θjxj,
j=0
where we append a bias feature x0 = 1. Given a training set {(x(i), y(i))}mi=1 , we choose θ to
minimize the squared-error cost
m
1Σ 2
J(θ) = hθ (x(i)) − y(i) .
2
i=1
Gradient descent updates θj := θj − α ∂J∂θ,j where for a single example
∂J
= h θ(x) − y xj .
∂θj
This is the LMS (least-mean-squares) update. The update is proportional to the error (hθ(x)−y),
which gives the algorithm its self-correcting character.
Batch gradient descent sums the gradient over all m examples; stochastic gradient descent
(SGD) updates after each example, which is cheaper and often faster on large datasets. The
learning rate α controls step size — too small and convergence is slow; too large and the cost
may diverge.
The normal equations give the closed-form minimizer. Stacking the data into a design matrix
⊤
X ∈ Rm×(n+1) (rows are x(i) ) and label vector y ∈ Rm, the optimal θ satisfies
∇θJ = X⊤(Xθ − y) = 0 =⇒ θ = (X⊤X)−1X⊤y.
This is the maximum-likelihood estimator under Gaussian noise y(i) | x(i) ∼ N (θ⊤x(i), σ2).
Key formula: normal equations
Closed-form least-squares solution. Requires X⊤X to be invertible (features linearly inde-
Worked mini-example. Two points: (x, y) = (1, 2), (2, 4), with bias feature. Then X =
1 1 2 3 5 −3
, y = 24 . Compute X⊤X = , X⊤y = 610 . Inverse of X⊤X is (det
1 2 3 5 −3 2
5·6−3·10
= 1), giving θ = −3·6+2·10 = 20 . So the line is h(x) = 0 + 2x = 2x, exactly fitting both points.
Practice Problem 1
{(1, 3), (2, 5), (3, 7)}. Compute the least-squares parameter vector θ (θ0, θ1) using the
normal equations. Verify that the residuals are zero.
