COMP 551 Applied Machine Learning 2025 new MIDTERM 2 Practice
Questions and answers McGill University
1 Lecture 1
1. Question: In the class, we have seen that both classification and regression are supervised learning
problems. What is the difference between these two supervised learning problems?
Answer: That classification is the problem of predicting a discrete class label output for an example.
yˆ ∈ {c1, ..., cn}
That regression is the problem of predicting a continuous quantity output for an example. yˆ ∈ R
2. Question: Give five real life applications for classification and five real life applications for regres-
sion.
Answer: Classification problems: Email Spam; Handwritten Digit Recognition; Image segmen-
tation; Speech Recognition; DNA Sequence Classification Regression problems: Housing Price
Prediction; Stock Price Prediction; ...
3. Question: When would you say that a particular model is a linear model?
Answer: The model is linear in terms of the parameter.
4. Question: What is the significance of w0 (bias) in the linear model: y = w0 + w1x? Why not use
a model like y = w1x?
Answer: If we do not add the bias term, we only cover the lines pass the origin.
2 Lecture 2
1. Question: What is overfitting? How can we find if a model is overfitting to a particular dataset?
Answer: Overfitting is the production of an analysis that corresponds too closely or exactly to a
particular set of data, and may therefore fail to fit additional data or predict future observations
reliably. (Like the m = 9 example in class)
We can use a separate test set to recognize overfitting.
2. Question: Suggest at least 3 approaches to solve the overfitting problem.
Answer:
Solution 1: Add more data points so that the model cannot overfit.
Solution 2: Regularization: Add a penalty term to the error function in order to discourage the
coefficient from reaching large. (In the case we cannot obtain more data.)
Solution 3: Model Selection: We can partition the data set into three parts, a training set, a
validation set and a test set. The training set is used for training the model to fit the data, the
validation set is used to measure differences in performance between models in order to select the
best one and the test set to assert that the model selection process does not overfit to the first two
sets.
3. Question: What is the hyperparameter of a model? How is it different from the parameters of the
model? How can we choose these hyperparameters?
Answer: a hyperparameter is a parameter whose value is set before the learning process begins. By
contrast, the parameters of the model are derived via training. (The regularization term in error
1
, function for linear regression model is the hyperparameter.)
Repeat training with different λs, and pick the one with the best perfomance in valid data set.
(Model Selection)
4. Question: While doing model selection, we choose the best hyperparameter based on the validation
set performance. What will happen if we choose the best hyperparameter based on training set
performance? What will happen if we choose the best hyperparameter based on the test set
performance? Do we really need a separate validation set?
Answer: Training set performance: Overfitting. The parameters you learn during training are
optimized to the training set. If you’re not careful, you can over-optimize the parameters, leading
to a model that’s really good on the training set.
Test set performance: The model has seen the test set so the test accuracy will not be proxy for
the real performance of the model.
If we use a set during your model fitting, the results we get on that set will not be fully indicative
of the general results we’ll obtain on completely new data. That’s why we hold out a fraction of
the data till the very end, past the point where we are making any decisions on what to do.
5. Question: What are the hyperparameters of the linear regression model? What are the hyperpa-
rameters of the k-NN classier?
Answer: Linear regression model: M(degree of polynomial) and regularization term (if exists)
K-NN classier: k; metric to compute NN (L1 vs. L2)
6. Question: Compare and contrast least squares approach and nearest neighbor approach in terms
of bias and variance.
Answer:
Decision boundary: very smooth (least squares) vs. wiggly and depends on a handful of input
points and their positions (nearest neighbors)
Stability: stable vs. less stable
Assumption: assumes the boundary is linear (strong) vs. does not have strong assumptions
Least squares: high bias, low variance
KNN: low bias, high variance
3 Lecture 3
1. Question: In the class, we have seen that if we use squared error loss, then the expected prediction
error is minimized by the conditional mean. Explain how nearest neighbor approach and least
squares approach are trying to approximate this conditional mean.
Answer:
Both k-nearest neighbors and least squares end up approximating conditional expectations by
averages. But they differ dramatically in terms of model assumptions.
Least squares assumes f(x) is well approximated by a globally linear function.
k-nearest neighbors assumes f(x) is well approximated by a locally constant function.
2. Question: What is Bayes rate? We have seen in the class that Bayes rate is the best possible
performance any classier can achieve. What does a classier require in order to achieve this optimal
error rate?
Answer:
The error rate of the Bayes classifier.
To achieve the optimal rate, we need to take a few data points first to learn the density distribution
first, and then proceed.
4 Lecture 4
1. Question: What is the advantage of using non-linear basis functions with a linear model like linear
regression?
2
, Answer: We are trying to predict y from x, for some future test case, but we are not trying to
model the distribution of x. Suppose also that we dont expect the best predictor for y to be a
linear function of x, so ordinary linear regression on the original variables wont work well. So we
need to allow for a non-linear function of x.
2. Question: What is the pseudo-inverse of a matrix? How is it different from the inverse of a matrix?
When will the psuedo-inverse and inverse be equivalent?
Answer:
A+ = (AT A)−1AT
Difference:
Recall the definitions of both inverse and pseudo-inverse of a matrix:
Let A ∈ Mn(R), then matrix B ∈ Mn(R) is called the inverse of A, if A · B = B · A = In, where
In is identity matrix of size n.
Let A ∈ Mn,m(R), then matrix B ∈ Mm,n(R), is called the pseudo-inverse of A, if the following 4
conditions are satisfied:
A·B·A=A
B·A·B=B
(AB)T = AB
(BA)T = BA
If A is square and invertible, then A+ = A−1.
3. Question: Explain the geometrical interpretation of least squares approach.
Answer:
https://ccrma.stanford.edu/~jos/sasp/Geometric_Interpretation_Least_Squares.html
4. Question: When can one resort to gradient descent to minimize the objective function?
Answer: The situation that it is not tractable to compute the analytical solution.
5. Question: What happens when the step size is too large in gradient descent? What happens when
the step size is too small?
Answer: Too small: the gradient descent can be slow.
Too large: gradient descent can overshoot the minimum. It may fail to converge, or even diverge.
6. Question: What is the difference between gradient descent and stochastic gradient descent?
Answer: Stochastic gradient descent: Approximate the gradient of this total loss by the gradient
of individual data point loss.
dlossk
w=w—α
dw
Here, k means k random data points.
The stochastic step only tries to approximate the true gradient. The approximation error can help
in escaping local minima.
Gradient descent:
wj = wj — α dL(w0, w1)
dwj
ΣN
i=1(w0 + w1x — y ) .
Here, L(w0, w1) = 2N 1 (i) (i) 2
In SGD, we are using the cost gradient of 1 example at each iteration, instead of using the sum of
the cost gradient of all examples.
7. Question: Gradient descent can always find the global minimum. True or False? If false, is there
any scenario when it is guaranteed to find the global minimum?
Answer: False. For non-linear models, gradient descent might get stuck in any of the local mini-
mum. It is guaranteed when the model is linear.
3
Questions and answers McGill University
1 Lecture 1
1. Question: In the class, we have seen that both classification and regression are supervised learning
problems. What is the difference between these two supervised learning problems?
Answer: That classification is the problem of predicting a discrete class label output for an example.
yˆ ∈ {c1, ..., cn}
That regression is the problem of predicting a continuous quantity output for an example. yˆ ∈ R
2. Question: Give five real life applications for classification and five real life applications for regres-
sion.
Answer: Classification problems: Email Spam; Handwritten Digit Recognition; Image segmen-
tation; Speech Recognition; DNA Sequence Classification Regression problems: Housing Price
Prediction; Stock Price Prediction; ...
3. Question: When would you say that a particular model is a linear model?
Answer: The model is linear in terms of the parameter.
4. Question: What is the significance of w0 (bias) in the linear model: y = w0 + w1x? Why not use
a model like y = w1x?
Answer: If we do not add the bias term, we only cover the lines pass the origin.
2 Lecture 2
1. Question: What is overfitting? How can we find if a model is overfitting to a particular dataset?
Answer: Overfitting is the production of an analysis that corresponds too closely or exactly to a
particular set of data, and may therefore fail to fit additional data or predict future observations
reliably. (Like the m = 9 example in class)
We can use a separate test set to recognize overfitting.
2. Question: Suggest at least 3 approaches to solve the overfitting problem.
Answer:
Solution 1: Add more data points so that the model cannot overfit.
Solution 2: Regularization: Add a penalty term to the error function in order to discourage the
coefficient from reaching large. (In the case we cannot obtain more data.)
Solution 3: Model Selection: We can partition the data set into three parts, a training set, a
validation set and a test set. The training set is used for training the model to fit the data, the
validation set is used to measure differences in performance between models in order to select the
best one and the test set to assert that the model selection process does not overfit to the first two
sets.
3. Question: What is the hyperparameter of a model? How is it different from the parameters of the
model? How can we choose these hyperparameters?
Answer: a hyperparameter is a parameter whose value is set before the learning process begins. By
contrast, the parameters of the model are derived via training. (The regularization term in error
1
, function for linear regression model is the hyperparameter.)
Repeat training with different λs, and pick the one with the best perfomance in valid data set.
(Model Selection)
4. Question: While doing model selection, we choose the best hyperparameter based on the validation
set performance. What will happen if we choose the best hyperparameter based on training set
performance? What will happen if we choose the best hyperparameter based on the test set
performance? Do we really need a separate validation set?
Answer: Training set performance: Overfitting. The parameters you learn during training are
optimized to the training set. If you’re not careful, you can over-optimize the parameters, leading
to a model that’s really good on the training set.
Test set performance: The model has seen the test set so the test accuracy will not be proxy for
the real performance of the model.
If we use a set during your model fitting, the results we get on that set will not be fully indicative
of the general results we’ll obtain on completely new data. That’s why we hold out a fraction of
the data till the very end, past the point where we are making any decisions on what to do.
5. Question: What are the hyperparameters of the linear regression model? What are the hyperpa-
rameters of the k-NN classier?
Answer: Linear regression model: M(degree of polynomial) and regularization term (if exists)
K-NN classier: k; metric to compute NN (L1 vs. L2)
6. Question: Compare and contrast least squares approach and nearest neighbor approach in terms
of bias and variance.
Answer:
Decision boundary: very smooth (least squares) vs. wiggly and depends on a handful of input
points and their positions (nearest neighbors)
Stability: stable vs. less stable
Assumption: assumes the boundary is linear (strong) vs. does not have strong assumptions
Least squares: high bias, low variance
KNN: low bias, high variance
3 Lecture 3
1. Question: In the class, we have seen that if we use squared error loss, then the expected prediction
error is minimized by the conditional mean. Explain how nearest neighbor approach and least
squares approach are trying to approximate this conditional mean.
Answer:
Both k-nearest neighbors and least squares end up approximating conditional expectations by
averages. But they differ dramatically in terms of model assumptions.
Least squares assumes f(x) is well approximated by a globally linear function.
k-nearest neighbors assumes f(x) is well approximated by a locally constant function.
2. Question: What is Bayes rate? We have seen in the class that Bayes rate is the best possible
performance any classier can achieve. What does a classier require in order to achieve this optimal
error rate?
Answer:
The error rate of the Bayes classifier.
To achieve the optimal rate, we need to take a few data points first to learn the density distribution
first, and then proceed.
4 Lecture 4
1. Question: What is the advantage of using non-linear basis functions with a linear model like linear
regression?
2
, Answer: We are trying to predict y from x, for some future test case, but we are not trying to
model the distribution of x. Suppose also that we dont expect the best predictor for y to be a
linear function of x, so ordinary linear regression on the original variables wont work well. So we
need to allow for a non-linear function of x.
2. Question: What is the pseudo-inverse of a matrix? How is it different from the inverse of a matrix?
When will the psuedo-inverse and inverse be equivalent?
Answer:
A+ = (AT A)−1AT
Difference:
Recall the definitions of both inverse and pseudo-inverse of a matrix:
Let A ∈ Mn(R), then matrix B ∈ Mn(R) is called the inverse of A, if A · B = B · A = In, where
In is identity matrix of size n.
Let A ∈ Mn,m(R), then matrix B ∈ Mm,n(R), is called the pseudo-inverse of A, if the following 4
conditions are satisfied:
A·B·A=A
B·A·B=B
(AB)T = AB
(BA)T = BA
If A is square and invertible, then A+ = A−1.
3. Question: Explain the geometrical interpretation of least squares approach.
Answer:
https://ccrma.stanford.edu/~jos/sasp/Geometric_Interpretation_Least_Squares.html
4. Question: When can one resort to gradient descent to minimize the objective function?
Answer: The situation that it is not tractable to compute the analytical solution.
5. Question: What happens when the step size is too large in gradient descent? What happens when
the step size is too small?
Answer: Too small: the gradient descent can be slow.
Too large: gradient descent can overshoot the minimum. It may fail to converge, or even diverge.
6. Question: What is the difference between gradient descent and stochastic gradient descent?
Answer: Stochastic gradient descent: Approximate the gradient of this total loss by the gradient
of individual data point loss.
dlossk
w=w—α
dw
Here, k means k random data points.
The stochastic step only tries to approximate the true gradient. The approximation error can help
in escaping local minima.
Gradient descent:
wj = wj — α dL(w0, w1)
dwj
ΣN
i=1(w0 + w1x — y ) .
Here, L(w0, w1) = 2N 1 (i) (i) 2
In SGD, we are using the cost gradient of 1 example at each iteration, instead of using the sum of
the cost gradient of all examples.
7. Question: Gradient descent can always find the global minimum. True or False? If false, is there
any scenario when it is guaranteed to find the global minimum?
Answer: False. For non-linear models, gradient descent might get stuck in any of the local mini-
mum. It is guaranteed when the model is linear.
3
