Statistical Learning
Made by Maxmillian Forman, please do not sell this document. Good Luck!
1. Bias-Variance Trade-off
1.1. Regression Function
Predictor Function Notation: is a predictor function, a prediction rule, it is an element of a set of all
possible predictor functions, denoted . We can subset with , hence .
Squared Loss Function: Given an independent test observation and a independent training sample
for a candidate prediction rule . The squared loss function is given by:
Population Risk. Also known as the expected loss, for a predictor function that is an element of all
possible predictor functions.
Regression Function: By minimizing the population risk we obtain the regression function which is
unknown in practice.
Statistical learning aims to estimate the fixed but unknown
Prediction Model: The regression function showcases the systematic information provides about
.
1.2. Empirical Risk
Empirical Risk Function: For a random sample and and sample prediction model
the empirical risk function for a candidate prediction rule is:
The empirical risk function is an estimator of the population risk .
A estimator for would be the minimum of the empirical risk.
Approximation Error: The approximation error of the risk function is:
For we want to find the smallest approximation error for , so we look at the greatest lower bound
of the approximation error.
A flexible model has a large set .
A large set corresponds to high variance of
, A high variance of results in high approximation error.
Overfitting: Refers to fitting to the training set too much, causing to predict empirical
risk and not the population risk. Occurring when a large set is present.
Overfitting occurs if we have a low training MSE and a high test MSE.
1.3. Regularization
Regularization: Refers to restricting all predictor functions to a smaller set , to control the bias
variance trade-off.
A large sample size corresponds to a low variance.
For :
As we increase :
Flexible model, risk of overfitting
Flexibility increases.
The variance of increases
The bias of decreases
As we decrease :
Inflexible model, risk of underfitting.
Flexibility decreases.
The variance of decreases
The bias of increases.
Bias: Bias refers to how far is from the true regression function .
Variance: Variance refers to the extent of fluctuations in .
Regularized Estimators: There are two estimators when we minimize on a smaller domain , regularized
and oracle estimators.
The regularized estimator estimates the oracle estimator, we are looking for the best prediction
rule in the subdomain .
1. Regularized Estimator:
Depends on the empirical risk function.
2. Oracle Estimator
Depends on the population risk function.
Population Mean Squared Error:
Sample Mean Squared Error:
Reducible and Irreducible Errors: Suppose is a possibly regularized estimate of , fitted using a
training set. For and from the test set we can rewrite the population MSE into a reducible error and
a irreducible error.
Bias-Variance Trade-off: We use the law of iterated expectations to decompose the reducible error into
the bias and variance of .
, Bias Variance for an Estimator: For an estimator of some parameter we can decompose the
mean squared error as:
1.4. Cross Validation
Hyper Parameter Optimization: We want to estimate the reducible error too find the optimal
subdomain for the estimator . So we introduce a hyperparameter where
which assists in finding the appropriate .
Validation Set Approach: A method to estimate . Outputs the validation set error rate which is an
estimate for test set error rate.
Advantages:
Relatively simple to understand
Disadvantages:
Validation set error rate, MSE, is variable due to random split.
Validation set error may overestimate test set error.
We use a subset of the initial dataset to train , so we loose some information.
1. Randomly splits the dataset into a training and validation dataset.
2. Trains on the training dataset.
3. The fitted estimator then predicts responses for the validation dataset.
4. We calculate the validation set error, MSE, which will serve as an approximation for test error
rate.
K-Fold Cross Validation: A method used to estimate . Outputs the validation set error rate which is an
estimate for test set error rate.
Advantages:
MSE variability is smaller then in validation set approach and LOOCV.
Disadvantages:
Has higher bias then LOOCV.
1. We divide the initial dataset into equally sized partitions.
2. We select partitions to be the training set and the remaining -th partition will be the
validation set.
3. We fit using the training set.
4. The fitted estimator then predicts responses for the -th validation dataset.
5. We calculate the validation set error and call it .
6. We repeat so for all partitions exclusively and compute the average
Leave One Out Cross Validation (LOOCV): A method to estimate . Outputs the validation set error rate
which is an estimate for test set error rate.
Advantages:
We use all the observations in the dataset, so LOOCV is less biased and does not
overestimate the test error rate.
Disadvantages:
Computationally Intensive
Has higher variance then -fold, but a lower bias.
1. We divide the dataset into a training and validation set where the validation set is of size
.
Made by Maxmillian Forman, please do not sell this document. Good Luck!
1. Bias-Variance Trade-off
1.1. Regression Function
Predictor Function Notation: is a predictor function, a prediction rule, it is an element of a set of all
possible predictor functions, denoted . We can subset with , hence .
Squared Loss Function: Given an independent test observation and a independent training sample
for a candidate prediction rule . The squared loss function is given by:
Population Risk. Also known as the expected loss, for a predictor function that is an element of all
possible predictor functions.
Regression Function: By minimizing the population risk we obtain the regression function which is
unknown in practice.
Statistical learning aims to estimate the fixed but unknown
Prediction Model: The regression function showcases the systematic information provides about
.
1.2. Empirical Risk
Empirical Risk Function: For a random sample and and sample prediction model
the empirical risk function for a candidate prediction rule is:
The empirical risk function is an estimator of the population risk .
A estimator for would be the minimum of the empirical risk.
Approximation Error: The approximation error of the risk function is:
For we want to find the smallest approximation error for , so we look at the greatest lower bound
of the approximation error.
A flexible model has a large set .
A large set corresponds to high variance of
, A high variance of results in high approximation error.
Overfitting: Refers to fitting to the training set too much, causing to predict empirical
risk and not the population risk. Occurring when a large set is present.
Overfitting occurs if we have a low training MSE and a high test MSE.
1.3. Regularization
Regularization: Refers to restricting all predictor functions to a smaller set , to control the bias
variance trade-off.
A large sample size corresponds to a low variance.
For :
As we increase :
Flexible model, risk of overfitting
Flexibility increases.
The variance of increases
The bias of decreases
As we decrease :
Inflexible model, risk of underfitting.
Flexibility decreases.
The variance of decreases
The bias of increases.
Bias: Bias refers to how far is from the true regression function .
Variance: Variance refers to the extent of fluctuations in .
Regularized Estimators: There are two estimators when we minimize on a smaller domain , regularized
and oracle estimators.
The regularized estimator estimates the oracle estimator, we are looking for the best prediction
rule in the subdomain .
1. Regularized Estimator:
Depends on the empirical risk function.
2. Oracle Estimator
Depends on the population risk function.
Population Mean Squared Error:
Sample Mean Squared Error:
Reducible and Irreducible Errors: Suppose is a possibly regularized estimate of , fitted using a
training set. For and from the test set we can rewrite the population MSE into a reducible error and
a irreducible error.
Bias-Variance Trade-off: We use the law of iterated expectations to decompose the reducible error into
the bias and variance of .
, Bias Variance for an Estimator: For an estimator of some parameter we can decompose the
mean squared error as:
1.4. Cross Validation
Hyper Parameter Optimization: We want to estimate the reducible error too find the optimal
subdomain for the estimator . So we introduce a hyperparameter where
which assists in finding the appropriate .
Validation Set Approach: A method to estimate . Outputs the validation set error rate which is an
estimate for test set error rate.
Advantages:
Relatively simple to understand
Disadvantages:
Validation set error rate, MSE, is variable due to random split.
Validation set error may overestimate test set error.
We use a subset of the initial dataset to train , so we loose some information.
1. Randomly splits the dataset into a training and validation dataset.
2. Trains on the training dataset.
3. The fitted estimator then predicts responses for the validation dataset.
4. We calculate the validation set error, MSE, which will serve as an approximation for test error
rate.
K-Fold Cross Validation: A method used to estimate . Outputs the validation set error rate which is an
estimate for test set error rate.
Advantages:
MSE variability is smaller then in validation set approach and LOOCV.
Disadvantages:
Has higher bias then LOOCV.
1. We divide the initial dataset into equally sized partitions.
2. We select partitions to be the training set and the remaining -th partition will be the
validation set.
3. We fit using the training set.
4. The fitted estimator then predicts responses for the -th validation dataset.
5. We calculate the validation set error and call it .
6. We repeat so for all partitions exclusively and compute the average
Leave One Out Cross Validation (LOOCV): A method to estimate . Outputs the validation set error rate
which is an estimate for test set error rate.
Advantages:
We use all the observations in the dataset, so LOOCV is less biased and does not
overestimate the test error rate.
Disadvantages:
Computationally Intensive
Has higher variance then -fold, but a lower bias.
1. We divide the dataset into a training and validation set where the validation set is of size
.
