CS7641 - MIDTERM UPDATED ACTUAL QUESTIONS AND
CORRECT ANSWERS
Question:
1. curse of dimensionality
Answer:
as the number of features or dimensions grows the amount of data we need to generalize accurate grows
exponentially
Question:
2. What is the key advantage of lazy learning
Answer:
instead of estimating the target function once for the entire space, they can estimate it locally and
differently for each new instance to be classified
Question:
3. What does learning consist of in instance-based algorithms?
Answer:
storing the data in O(n) time
Question:
4. Why is constructing a different approximation to the target function for each distinct query distance
good?
Answer:
When the target function is very complex but still can be described by a collection of less complex local
approximation
Question:
5. What are the disadvantages of instance-based learning?
Answer:
- cost of classifying a new instance is high: takes O(n) time to classify an example
- consider all the attributes of the instances when attempting to retrieve similar training examples from
memory. If the target concept depends on only a few of the many attributes, then the instances that are
truly most similar may be a large distance apart
Question:
6. Describe the KNN algorithm.
Answer:
- assumes all instances correspond to points in n-dimensional space
- nearest neighbors are defined in terms of some distance function
- training: for each training example store it
- classification: 1) given a query instance find the k closes training examples 2) find the most common
label among the k instances
,Question:
7. In KNN, when all the training examples are considered what is it called
Answer:
a global method
Question:
8. In KNN, when only the nearest training examples are considered what is it called
Answer:
a local method
Question:
9. To not have to worry about k in KNN, what can we use?
Answer:
a distance-weighting function.
Question:
10. why does overfitting tend to occur during later iterations but not during earlier iterations of
backpropagation?
Answer:
- the complexity of the hypotheses that can be reached by backpropagation increases with the number of
weight-tuning iterations
- given enough weight-tuning iterations, backpropagation will often be able to create overly complex
decision surfaces that fits noise in the training data
Question:
11. What is the curse of dimensionality in KNN
Answer:
Distance between instances is calculated based on all attributes of the instance which can be problematic
when only a subset of features are relevant for classification. On the relevant features, they could be close
or identical but on others they may be far apart so their distance would be large
Question:
12. What is a solution to the curse of dimensionality in KNN?
Answer:
Option 1: weight each attribute differently when calculating the distance between two instances. Option 2:
completely eliminate the least relevant attributes from the instance space using LOOCV
Question:
13. What is lazy learning?
Answer:
When you defer the decision of how to generalize beyond the training data until each new query instance is
encountered?
Question:
14. What is eager learning?
Answer:
Generalized beyond the training data before observing the new query.
,Question:
15. What are the differences between lazy and eager learning as it relates to computation time.
Answer:
Lazy learners will require less computation during training but more computation when they must predict
the target value for a new query
Question:
16. Describe the differences in the inductive bias of lazy vs eager learners?
Answer:
Lazy methods may consider the new query instance when deciding how to generalize beyond the training
data while methods cannot consider the new query instance because they have already chosen their global
approximation to the target function. This affects the accuracy of the learner because the eager learner
must commit to a single hypothesis instead of using many different local approximations like the lazy
learner.
Question:
17. Why is RBF a nice middle ground between eager and lazy learners?
Answer:
It commits to a global approximation to the target function at training time; however it represents the
global function as a linear combination of multiple local kernel functions.
Question:
18. When k is equal to the number of training examples, what will the probability vector be?
Answer:
It will be equal to the prior distribution over the examples
Question:
19. What happens to the number of different predictions KNN can make as k increases.
Answer:
It initially increases then decreases.
Question:
20. What happens to the bias and variance of different predictions KNN can make as k increases.
Answer:
the bias increases and then the variance decreases
Question:
21. what are the properties of a nearest-neighbor classifier
Answer:
- can perfectly classify the training set unless identical instances with different labels are included
- by choosing the right examples, it can more or less represent any decision boundary or at least an
arbitrarily close piecewise linear approximation
- has low bias but high variance
- will overfit if the training data is limited, noise or unrepresentative
Question:
22. Define preference bias
, Answer:
our belief about what makes a good hypothesis.
Question:
23. What is KNN's preference bias?
Answer:
- locality: near points are similar
- smoothness: by choosing to average/vote we are expecting functions to behave smoothly
- all features matter equally
Question:
24. How do we inject domain knowledge into KNN?
Answer:
By defining a distance function. There is a best distance function for each problem. We just don't know it.
(No Free Lunch)
Question:
25. What does boosting reduce bias or variance?
Answer:
Boosting is a variance reduction technique
Question:
26. What are three reasons why a model will misclassify a test instance?
Answer:
- if instances from different classes are described by the same feature vectors
- the model lacks expressivity to exactly represent the target concept (high bias)
- variance
Question:
27. Why do tree models have high variance?
Answer:
If we change the training data sufficiently for another feature to be selected at the root of the tree, then the
rest of the tree is likely to be different as well
Question:
28. What is boosting effective in doing? And what is the effect of this?
Answer:
Increasing the margin of examples even if they are already on the correct side of the decision boundary.
Boosting may continue to improve performance on the test set even after the training error has been
reduced to zero
Question:
29. what is weak learnability
Answer:
they hypothesis is slightly better than chance.
CORRECT ANSWERS
Question:
1. curse of dimensionality
Answer:
as the number of features or dimensions grows the amount of data we need to generalize accurate grows
exponentially
Question:
2. What is the key advantage of lazy learning
Answer:
instead of estimating the target function once for the entire space, they can estimate it locally and
differently for each new instance to be classified
Question:
3. What does learning consist of in instance-based algorithms?
Answer:
storing the data in O(n) time
Question:
4. Why is constructing a different approximation to the target function for each distinct query distance
good?
Answer:
When the target function is very complex but still can be described by a collection of less complex local
approximation
Question:
5. What are the disadvantages of instance-based learning?
Answer:
- cost of classifying a new instance is high: takes O(n) time to classify an example
- consider all the attributes of the instances when attempting to retrieve similar training examples from
memory. If the target concept depends on only a few of the many attributes, then the instances that are
truly most similar may be a large distance apart
Question:
6. Describe the KNN algorithm.
Answer:
- assumes all instances correspond to points in n-dimensional space
- nearest neighbors are defined in terms of some distance function
- training: for each training example store it
- classification: 1) given a query instance find the k closes training examples 2) find the most common
label among the k instances
,Question:
7. In KNN, when all the training examples are considered what is it called
Answer:
a global method
Question:
8. In KNN, when only the nearest training examples are considered what is it called
Answer:
a local method
Question:
9. To not have to worry about k in KNN, what can we use?
Answer:
a distance-weighting function.
Question:
10. why does overfitting tend to occur during later iterations but not during earlier iterations of
backpropagation?
Answer:
- the complexity of the hypotheses that can be reached by backpropagation increases with the number of
weight-tuning iterations
- given enough weight-tuning iterations, backpropagation will often be able to create overly complex
decision surfaces that fits noise in the training data
Question:
11. What is the curse of dimensionality in KNN
Answer:
Distance between instances is calculated based on all attributes of the instance which can be problematic
when only a subset of features are relevant for classification. On the relevant features, they could be close
or identical but on others they may be far apart so their distance would be large
Question:
12. What is a solution to the curse of dimensionality in KNN?
Answer:
Option 1: weight each attribute differently when calculating the distance between two instances. Option 2:
completely eliminate the least relevant attributes from the instance space using LOOCV
Question:
13. What is lazy learning?
Answer:
When you defer the decision of how to generalize beyond the training data until each new query instance is
encountered?
Question:
14. What is eager learning?
Answer:
Generalized beyond the training data before observing the new query.
,Question:
15. What are the differences between lazy and eager learning as it relates to computation time.
Answer:
Lazy learners will require less computation during training but more computation when they must predict
the target value for a new query
Question:
16. Describe the differences in the inductive bias of lazy vs eager learners?
Answer:
Lazy methods may consider the new query instance when deciding how to generalize beyond the training
data while methods cannot consider the new query instance because they have already chosen their global
approximation to the target function. This affects the accuracy of the learner because the eager learner
must commit to a single hypothesis instead of using many different local approximations like the lazy
learner.
Question:
17. Why is RBF a nice middle ground between eager and lazy learners?
Answer:
It commits to a global approximation to the target function at training time; however it represents the
global function as a linear combination of multiple local kernel functions.
Question:
18. When k is equal to the number of training examples, what will the probability vector be?
Answer:
It will be equal to the prior distribution over the examples
Question:
19. What happens to the number of different predictions KNN can make as k increases.
Answer:
It initially increases then decreases.
Question:
20. What happens to the bias and variance of different predictions KNN can make as k increases.
Answer:
the bias increases and then the variance decreases
Question:
21. what are the properties of a nearest-neighbor classifier
Answer:
- can perfectly classify the training set unless identical instances with different labels are included
- by choosing the right examples, it can more or less represent any decision boundary or at least an
arbitrarily close piecewise linear approximation
- has low bias but high variance
- will overfit if the training data is limited, noise or unrepresentative
Question:
22. Define preference bias
, Answer:
our belief about what makes a good hypothesis.
Question:
23. What is KNN's preference bias?
Answer:
- locality: near points are similar
- smoothness: by choosing to average/vote we are expecting functions to behave smoothly
- all features matter equally
Question:
24. How do we inject domain knowledge into KNN?
Answer:
By defining a distance function. There is a best distance function for each problem. We just don't know it.
(No Free Lunch)
Question:
25. What does boosting reduce bias or variance?
Answer:
Boosting is a variance reduction technique
Question:
26. What are three reasons why a model will misclassify a test instance?
Answer:
- if instances from different classes are described by the same feature vectors
- the model lacks expressivity to exactly represent the target concept (high bias)
- variance
Question:
27. Why do tree models have high variance?
Answer:
If we change the training data sufficiently for another feature to be selected at the root of the tree, then the
rest of the tree is likely to be different as well
Question:
28. What is boosting effective in doing? And what is the effect of this?
Answer:
Increasing the margin of examples even if they are already on the correct side of the decision boundary.
Boosting may continue to improve performance on the test set even after the training error has been
reduced to zero
Question:
29. what is weak learnability
Answer:
they hypothesis is slightly better than chance.
