SOA EXAM PA QUESTIONS AND
ANSWERS
Describe the Bias-Variance trade off. - Correct Answers -The bias-variance trade off
relates to making your model a good balance of being fitted to the training data and
being predictive of the test data. When a model is fitted too well (too complex), then it
tends to lack predictive power. When a model predicts too well, it tends to not be fitted
too well to the training data.
Bias refers to the ability to capture the signal in the data. It is the part of the test error
caused by a model that is not flexible enough to capture the underlying signal.
Variance refers to the ability to capture the predictive performance. We lower variance
by making the model a better fit to future predictions.
Describe stepwise selection. - Correct Answers -Stepwise selection is a process of
feature selection where we either start with a null model (forward selection) or start with
the full model (backwards selection) and add or remove one variable at a time,
depending on which variable causes the best improvement in the model by adding or
dropping it.
Define regularization. - Correct Answers -Regularization is a method for reducing the
complexity of a linear model. We consider a model that hosts all of the possible
features, and we shrink the coefficient estimates towards zero for those features that
have limited predictive power. This is an alternative to completely removing the
features.
Define interaction. - Correct Answers -Interaction arises when the expected effect of
one predictor on the target variable depends on the value (or level) of another predictor.
Describe what pruning does. - Correct Answers -Pruning is a method of feature
selection in decision trees. It involves removing branches of the tree that lack in
predictive performance, subject to a control parameter.
Describe PCA. - Correct Answers -Principal component analysis is a data analytic
technique that transforms high dimensional datasets into a smaller, more manageable
set of representative variables that capture most of the information in the original
dataset. It combines multiple variables into new features called Principal Components
(PCs) which capture multiple variables into one.
,Describe cluster analysis. - Correct Answers -Cluster analysis is a data analytic tool for
high dimensional datasets. It works by partitioning observations into a set of distinct
groups, known as clusters. The goal is to uncover interesting subgroups in the dataset.
What is the formula for AIC? - Correct Answers -AIC = -2*l + 2*p
l = maximized loglikelihood of the model on the training set
p = number of parameters
What is the formula for BIC? - Correct Answers -BIC = -2*l + p*ln(n)
l = maximized loglikelihood of the model on the training set
n = number of observations in the training set
Describe the pros and cons of using AIC in stepwise selection. - Correct Answers -The
AIC penalty parameter is lower than that of BIC, so the AIC is less stringent than BIC
and leads to a more complex model. AIC is normally combined with backwards
stepwise selection.
Describe the pros and cons of using BIC in stepwise selection. - Correct Answers -The
BIC penalty parameter is higher than that of AIC, so the BIC is more stringent than AIC
and leads to a less complex model. BIC is normally combined with forwards stepwise
selection.
Describe the pros and cons of using forward stepwise selection. - Correct Answers -
Forward stepwise selection involves starting with the null model (only the intercept) and
adding one variable at a time until we no longer want to add anymore. We add the
variable that leads to the greatest improvement in the model, according to a pre-
specified performance metric.
It tends to lead to a much smaller (less complex) and interpretable model. However, it
can be underfitted in some regards.
Describe the pros and cons of using backwards stepwise selection. - Correct Answers -
Backwards stepwise selection involves starting with the full model (all variables in the
dataset are accounted for), and dropping one variable at a time, depending on which
variable being dropped leads to the greatest improvement of model performance,
according to a pre-specified performance metric.
It tends to lead to a more complex model than forward selection, which can sometimes
be overfitted still.
Discuss the pros and cons of using a random forest. - Correct Answers -Pros: Relative
to a single tree, random forests are usually much more robust. Combining all B of the
, base trees contributed substantial variance reduction, which results in much more
precise predictions.
Cons: Random forests are generally not as interpretable as base trees, and it takes a
considerably longer time to implement a random forest compared to a decision tree due
to the computational burden that comes with it
Describe the pros and cons of decision trees. - Correct Answers -Pros: It is easy to
interpret and explain to non-technical audiences. It can also be displayed graphically. It
does a good job of modeling non-linear relationships and automatically recognizes
interactions. Categorical predictors are automatically handled without binarization and
variables are automatically selected.
Cons: Decision trees are more prone to overfitting than GLMs, even with pruning. In
order to capture the effects of a numeric variable, we need to split based on this
variable repeatedly, which gives rise to a complex tree with large depth. There is also a
lack of model diagnostic tools.
Describe the pros and cons of boosted trees. - Correct Answers -Pros: Boosted trees
often perform better in terms of prediction accuracy than random forests, due to their
emphasis on bias reduction.
Cons: Boosted trees are more prone to overfitting than random forests. There is also a
loss of interpretability when compared to base trees.
Describe random forests. - Correct Answers -Random forests entail generating multiple
bootstrapped (with replacement) samples of the training set and fitting base trees in
parallel, independently on each of the samples. We combine all of these base trees to
form an overall prediction. In each sample, a random sample of m predictors is chosen
as the split candidates of the p available features.
Describe boosted trees. - Correct Answers -Boosting builds a sequence of
interdependent trees using information from previously grown trees. In each iteration,
we fit a tree to the residuals of the preceding tree, and a scaled down version of the
current tree's predictions is subtracted from the preceding tree's residuals to form the
new residuals. The whole process is repeated, with the effect being that each tree will
focus on predicting observations that the previous tree predicted poorly.
How can we tell how many PCs to use in PCA? - Correct Answers -We can use a scree
plot, which is a plot of the principle component and the proportion of variance explained.
We choose the number of PCs by eyeballing the plot too look for the point at which the
PVE of a subsequent PC has dropped off to a sufficiently low level, which is known as
the elbow.
How can we tell how many PCs to use in Cluster Analysis? - Correct Answers -We use
the elbow method. To do this, we need to compute the ratio of between-cluster sum of
ANSWERS
Describe the Bias-Variance trade off. - Correct Answers -The bias-variance trade off
relates to making your model a good balance of being fitted to the training data and
being predictive of the test data. When a model is fitted too well (too complex), then it
tends to lack predictive power. When a model predicts too well, it tends to not be fitted
too well to the training data.
Bias refers to the ability to capture the signal in the data. It is the part of the test error
caused by a model that is not flexible enough to capture the underlying signal.
Variance refers to the ability to capture the predictive performance. We lower variance
by making the model a better fit to future predictions.
Describe stepwise selection. - Correct Answers -Stepwise selection is a process of
feature selection where we either start with a null model (forward selection) or start with
the full model (backwards selection) and add or remove one variable at a time,
depending on which variable causes the best improvement in the model by adding or
dropping it.
Define regularization. - Correct Answers -Regularization is a method for reducing the
complexity of a linear model. We consider a model that hosts all of the possible
features, and we shrink the coefficient estimates towards zero for those features that
have limited predictive power. This is an alternative to completely removing the
features.
Define interaction. - Correct Answers -Interaction arises when the expected effect of
one predictor on the target variable depends on the value (or level) of another predictor.
Describe what pruning does. - Correct Answers -Pruning is a method of feature
selection in decision trees. It involves removing branches of the tree that lack in
predictive performance, subject to a control parameter.
Describe PCA. - Correct Answers -Principal component analysis is a data analytic
technique that transforms high dimensional datasets into a smaller, more manageable
set of representative variables that capture most of the information in the original
dataset. It combines multiple variables into new features called Principal Components
(PCs) which capture multiple variables into one.
,Describe cluster analysis. - Correct Answers -Cluster analysis is a data analytic tool for
high dimensional datasets. It works by partitioning observations into a set of distinct
groups, known as clusters. The goal is to uncover interesting subgroups in the dataset.
What is the formula for AIC? - Correct Answers -AIC = -2*l + 2*p
l = maximized loglikelihood of the model on the training set
p = number of parameters
What is the formula for BIC? - Correct Answers -BIC = -2*l + p*ln(n)
l = maximized loglikelihood of the model on the training set
n = number of observations in the training set
Describe the pros and cons of using AIC in stepwise selection. - Correct Answers -The
AIC penalty parameter is lower than that of BIC, so the AIC is less stringent than BIC
and leads to a more complex model. AIC is normally combined with backwards
stepwise selection.
Describe the pros and cons of using BIC in stepwise selection. - Correct Answers -The
BIC penalty parameter is higher than that of AIC, so the BIC is more stringent than AIC
and leads to a less complex model. BIC is normally combined with forwards stepwise
selection.
Describe the pros and cons of using forward stepwise selection. - Correct Answers -
Forward stepwise selection involves starting with the null model (only the intercept) and
adding one variable at a time until we no longer want to add anymore. We add the
variable that leads to the greatest improvement in the model, according to a pre-
specified performance metric.
It tends to lead to a much smaller (less complex) and interpretable model. However, it
can be underfitted in some regards.
Describe the pros and cons of using backwards stepwise selection. - Correct Answers -
Backwards stepwise selection involves starting with the full model (all variables in the
dataset are accounted for), and dropping one variable at a time, depending on which
variable being dropped leads to the greatest improvement of model performance,
according to a pre-specified performance metric.
It tends to lead to a more complex model than forward selection, which can sometimes
be overfitted still.
Discuss the pros and cons of using a random forest. - Correct Answers -Pros: Relative
to a single tree, random forests are usually much more robust. Combining all B of the
, base trees contributed substantial variance reduction, which results in much more
precise predictions.
Cons: Random forests are generally not as interpretable as base trees, and it takes a
considerably longer time to implement a random forest compared to a decision tree due
to the computational burden that comes with it
Describe the pros and cons of decision trees. - Correct Answers -Pros: It is easy to
interpret and explain to non-technical audiences. It can also be displayed graphically. It
does a good job of modeling non-linear relationships and automatically recognizes
interactions. Categorical predictors are automatically handled without binarization and
variables are automatically selected.
Cons: Decision trees are more prone to overfitting than GLMs, even with pruning. In
order to capture the effects of a numeric variable, we need to split based on this
variable repeatedly, which gives rise to a complex tree with large depth. There is also a
lack of model diagnostic tools.
Describe the pros and cons of boosted trees. - Correct Answers -Pros: Boosted trees
often perform better in terms of prediction accuracy than random forests, due to their
emphasis on bias reduction.
Cons: Boosted trees are more prone to overfitting than random forests. There is also a
loss of interpretability when compared to base trees.
Describe random forests. - Correct Answers -Random forests entail generating multiple
bootstrapped (with replacement) samples of the training set and fitting base trees in
parallel, independently on each of the samples. We combine all of these base trees to
form an overall prediction. In each sample, a random sample of m predictors is chosen
as the split candidates of the p available features.
Describe boosted trees. - Correct Answers -Boosting builds a sequence of
interdependent trees using information from previously grown trees. In each iteration,
we fit a tree to the residuals of the preceding tree, and a scaled down version of the
current tree's predictions is subtracted from the preceding tree's residuals to form the
new residuals. The whole process is repeated, with the effect being that each tree will
focus on predicting observations that the previous tree predicted poorly.
How can we tell how many PCs to use in PCA? - Correct Answers -We can use a scree
plot, which is a plot of the principle component and the proportion of variance explained.
We choose the number of PCs by eyeballing the plot too look for the point at which the
PVE of a subsequent PC has dropped off to a sufficiently low level, which is known as
the elbow.
How can we tell how many PCs to use in Cluster Analysis? - Correct Answers -We use
the elbow method. To do this, we need to compute the ratio of between-cluster sum of
