10/15/24, 9:24 Super cheatsheet machine
AM learning
CS 229 – Machine Learning https://stanford.edu/~shervine
Super VIP Cheatsheet: Machine Learning 3.1
3.2
Neural Networks.............................................................................................................8
Convolutional Neural Networks...................................................................................8
4
Afshine AMiDi and Shervine AMiDi
October 6, 2018
Contents
1 Su per vised Learning 2
1.1 Introduction to Supervised Learning..........................................................................2
1.2 Notations and general concepts...................................................................................2
1.3 Linear models..................................................................................................................2
1.3.1 Linear regression..............................................................................................2
1.3.2 Classification and logistic regression............................................................3
1.3.3 Generalized Linear Models.............................................................................3
1.4 Support Vector Machines..............................................................................................3
1.5 Generative Learning.......................................................................................................4
1.5.1 Gaussian Discriminant Analysis....................................................................4
1.5.2 Naive Bayes.......................................................................................................4
1.6 Tree-based and ensemble methods..............................................................................4
1.7 Other non-parametric approaches...............................................................................4
1.8 Learning Theory..............................................................................................................5
2 Unsupervised Learning 6
2.1 Introduction to Unsupervised Learning.....................................................................6
2.2 Clustering.........................................................................................................................6
2.2.1 Expectation-Maximization..............................................................................6
2.2.2 k-means clustering............................................................................................6
2.2.3 Hierarchical clustering.....................................................................................6
2.2.4 Clustering assessment metrics........................................................................6
2.3 Dimension reduction......................................................................................................7
2.3.1 Principal component analysis........................................................................7
2.3.2 Independent component analysis..................................................................7
3 Deep Learning 8
about:blan 1/
k 40
,10/15/24, 9:24 Super cheatsheet machine
Machine
AM Learning Tips and Tricks 10 learning
4.1 Metrics............................................................................................................................10
4.1.1 Classification....................................................................................................10
4.1.2 Regression........................................................................................................10
4.2 Model selection..............................................................................................................11
4.3 Diagnostics.....................................................................................................................11
5 Refreshers 12
5.1 Probabilities and Statistics..........................................................................................12
5.1.1 Introduction to Probability and Combinatorics.......................................12
5.1.2 Conditional Probability.................................................................................12
5.1.3 Random Variables..........................................................................................13
5.1.4 Jointly Distributed Random Variables.......................................................13
5.1.5 Parameter estimation.....................................................................................14
5.2 Linear Algebra and Calculus......................................................................................14
5.2.1 General notations...........................................................................................14
5.2.2 Matrix operations...........................................................................................15
5.2.3 Matrix properties............................................................................................15
5.2.4 Matrix calculus...............................................................................................16
about:blan 2/
k 40
,10/15/24, 9:24 Super cheatsheet machine
AM learning
Recurrent Neural Networks8
Reinforcement Learning and Control9
STanFOrD UniversiTY1 FaLL
2018
about:blan 3/
k 40
, 10/15/24, 9:24 Super cheatsheet machine
AM learning
CS 229 – Machine Learning Shervine AMiDi & AFshine
AMiDi
1 Su per vised Learning
❒ Co st fun ction – The cost function J is commonly used to assess the performance of a
model, and is defined with the loss function L as follows:
1.1 Intr odu ction to Su per vised Learning
X
m
Given a set of data points {x(1) , ..., x(m)} associated to a set of outcomes {y(1) , ..., y(m)}, J (θ) = L(hθ (x
(i)
), y
(i)
)
we want to build a classifier that learns how to predict y from x.
i=1
❒ Type of pr e dictio n – The different types of predictive models are summed up in the
table below:
❒ Gr a dien t desc ent – By noting α ∈ R the learning rate, the update rule for gradient
Regression Cla ssifi er descent is expressed with the learning rate and the cost function J as follows:
Outcom e Continuous Class θ →− θ − α∇J (θ)
Exa m ple s Linear regression Logistic regression, SVM, Naive Bayes
❒ Type o f m o del – The different models are summed up in the table below:
Discr im in a tive m odel Gen er a tive m odel
Goal Directly estimate P (y|x) Estimate P (x|y) to deduce P (y|x)
W h a t’s lear ned Decision boundary Probability distributions of the data
Remark: Stochastic gradient descent (SGD) is updating the parameter based on each training
Illustr ation example, and batch gradient descent is on a batch of training examples.
❒ Like lih o od – The likelihood of a model L(θ) given parameters θ is used to find the optimal
parameters θ through maximizing the likelihood. In practice, we use the log-likelihood ℓ(θ) =
log(L(θ)) which is easier to optimize. We have:
Exam ples Regressions, SVMs GDA, Naive Bayes opt
θ = arg max L(θ)
θ
1.2 Notations and gener al concepts
❒ New to n’s algo r ith m – The Newton’s algorithm is a numerical method that finds θ such
❒ Hypo th esis – The hypothesis is noted hθ and is the model that we choose. For a given that ℓ′(θ) = 0. Its update rule is as follows:
(i) (i)
input data x , the model prediction output is hθ (x ). ℓ′(θ)
θ→θ−
❒ L o ss fu nctio n – A loss function is a function L : (z,y) ∈ R × Y −→ L(z,y) ∈ R that takes as ℓ′′ (θ)
inputs the predicted value z corresponding to the real data value y and outputs how different Remark: the multidimensional generalization, also known as the Newton-Raphson method, has
they are. The common loss functions are summed up in the table below: the following update rule: 2 −1
θ → θ − ∇ ℓ(θ) ∇ ℓ(θ)
L ea st squared Logistic Hinge Cr o ss-entropy θ θ
1 2
(y − z) log(1 + exp(−yz)) max(0,1 − yz) −y log(z) + (1 − y) log(1 − z)
2
1.3 Linear m odels
about:blan 4/
k 40
AM learning
CS 229 – Machine Learning https://stanford.edu/~shervine
Super VIP Cheatsheet: Machine Learning 3.1
3.2
Neural Networks.............................................................................................................8
Convolutional Neural Networks...................................................................................8
4
Afshine AMiDi and Shervine AMiDi
October 6, 2018
Contents
1 Su per vised Learning 2
1.1 Introduction to Supervised Learning..........................................................................2
1.2 Notations and general concepts...................................................................................2
1.3 Linear models..................................................................................................................2
1.3.1 Linear regression..............................................................................................2
1.3.2 Classification and logistic regression............................................................3
1.3.3 Generalized Linear Models.............................................................................3
1.4 Support Vector Machines..............................................................................................3
1.5 Generative Learning.......................................................................................................4
1.5.1 Gaussian Discriminant Analysis....................................................................4
1.5.2 Naive Bayes.......................................................................................................4
1.6 Tree-based and ensemble methods..............................................................................4
1.7 Other non-parametric approaches...............................................................................4
1.8 Learning Theory..............................................................................................................5
2 Unsupervised Learning 6
2.1 Introduction to Unsupervised Learning.....................................................................6
2.2 Clustering.........................................................................................................................6
2.2.1 Expectation-Maximization..............................................................................6
2.2.2 k-means clustering............................................................................................6
2.2.3 Hierarchical clustering.....................................................................................6
2.2.4 Clustering assessment metrics........................................................................6
2.3 Dimension reduction......................................................................................................7
2.3.1 Principal component analysis........................................................................7
2.3.2 Independent component analysis..................................................................7
3 Deep Learning 8
about:blan 1/
k 40
,10/15/24, 9:24 Super cheatsheet machine
Machine
AM Learning Tips and Tricks 10 learning
4.1 Metrics............................................................................................................................10
4.1.1 Classification....................................................................................................10
4.1.2 Regression........................................................................................................10
4.2 Model selection..............................................................................................................11
4.3 Diagnostics.....................................................................................................................11
5 Refreshers 12
5.1 Probabilities and Statistics..........................................................................................12
5.1.1 Introduction to Probability and Combinatorics.......................................12
5.1.2 Conditional Probability.................................................................................12
5.1.3 Random Variables..........................................................................................13
5.1.4 Jointly Distributed Random Variables.......................................................13
5.1.5 Parameter estimation.....................................................................................14
5.2 Linear Algebra and Calculus......................................................................................14
5.2.1 General notations...........................................................................................14
5.2.2 Matrix operations...........................................................................................15
5.2.3 Matrix properties............................................................................................15
5.2.4 Matrix calculus...............................................................................................16
about:blan 2/
k 40
,10/15/24, 9:24 Super cheatsheet machine
AM learning
Recurrent Neural Networks8
Reinforcement Learning and Control9
STanFOrD UniversiTY1 FaLL
2018
about:blan 3/
k 40
, 10/15/24, 9:24 Super cheatsheet machine
AM learning
CS 229 – Machine Learning Shervine AMiDi & AFshine
AMiDi
1 Su per vised Learning
❒ Co st fun ction – The cost function J is commonly used to assess the performance of a
model, and is defined with the loss function L as follows:
1.1 Intr odu ction to Su per vised Learning
X
m
Given a set of data points {x(1) , ..., x(m)} associated to a set of outcomes {y(1) , ..., y(m)}, J (θ) = L(hθ (x
(i)
), y
(i)
)
we want to build a classifier that learns how to predict y from x.
i=1
❒ Type of pr e dictio n – The different types of predictive models are summed up in the
table below:
❒ Gr a dien t desc ent – By noting α ∈ R the learning rate, the update rule for gradient
Regression Cla ssifi er descent is expressed with the learning rate and the cost function J as follows:
Outcom e Continuous Class θ →− θ − α∇J (θ)
Exa m ple s Linear regression Logistic regression, SVM, Naive Bayes
❒ Type o f m o del – The different models are summed up in the table below:
Discr im in a tive m odel Gen er a tive m odel
Goal Directly estimate P (y|x) Estimate P (x|y) to deduce P (y|x)
W h a t’s lear ned Decision boundary Probability distributions of the data
Remark: Stochastic gradient descent (SGD) is updating the parameter based on each training
Illustr ation example, and batch gradient descent is on a batch of training examples.
❒ Like lih o od – The likelihood of a model L(θ) given parameters θ is used to find the optimal
parameters θ through maximizing the likelihood. In practice, we use the log-likelihood ℓ(θ) =
log(L(θ)) which is easier to optimize. We have:
Exam ples Regressions, SVMs GDA, Naive Bayes opt
θ = arg max L(θ)
θ
1.2 Notations and gener al concepts
❒ New to n’s algo r ith m – The Newton’s algorithm is a numerical method that finds θ such
❒ Hypo th esis – The hypothesis is noted hθ and is the model that we choose. For a given that ℓ′(θ) = 0. Its update rule is as follows:
(i) (i)
input data x , the model prediction output is hθ (x ). ℓ′(θ)
θ→θ−
❒ L o ss fu nctio n – A loss function is a function L : (z,y) ∈ R × Y −→ L(z,y) ∈ R that takes as ℓ′′ (θ)
inputs the predicted value z corresponding to the real data value y and outputs how different Remark: the multidimensional generalization, also known as the Newton-Raphson method, has
they are. The common loss functions are summed up in the table below: the following update rule: 2 −1
θ → θ − ∇ ℓ(θ) ∇ ℓ(θ)
L ea st squared Logistic Hinge Cr o ss-entropy θ θ
1 2
(y − z) log(1 + exp(−yz)) max(0,1 − yz) −y log(z) + (1 − y) log(1 − z)
2
1.3 Linear m odels
about:blan 4/
k 40
