SOA Exam PA
Describe the Bias-Variance trade off.
ANSWER The bias-variance trade off relates to making your model a good balance of being
fitted to the training data and being predictive of the test data. When a model is fitted too well
(too complex), then it tends to lack predictive power. When a model predicts too well, it tends
to not be fitted too well to the training data.
Bias refers to the ability to capture the signal in the data. It is the part of the test error caused
by a model that is not flexible enough to capture the underlying signal.
Variance refers to the ability to capture the predictive performance. We lower variance by
making the model a better fit to future predictions.
Describe stepwise selection.
ANSWER Stepwise selection is a process of feature selection where we either start with a null
model (forward selection) or start with the full model (backwards selection) and add or remove
one variable at a time, depending on which variable causes the best improvement in the model
by adding or dropping it.
Define regularization.
ANSWER Regularization is a method for reducing the complexity of a linear model. We
consider a model that hosts all of the possible features, and we shrink the coefficient estimates
towards zero for those features that have limited predictive power. This is an alternative to
completely removing the features.
Define interaction.
ANSWER Interaction arises when the expected effect of one predictor on the target variable
depends on the value (or level) of another predictor.
Describe what pruning does.
ANSWER Pruning is a method of feature selection in decision trees. It involves removing
branches of the tree that lack in predictive performance, subject to a control parameter.
Describe PCA.
ANSWER Principal component analysis is a data analytic technique that transforms high
dimensional datasets into a smaller, more manageable set of representative variables that
capture most of the information in the original dataset. It combines multiple variables into new
features called Principal Components (PCs) which capture multiple variables into one.
Describe cluster analysis.
, ANSWER Cluster analysis is a data analytic tool for high dimensional datasets. It works by
partitioning observations into a set of distinct groups, known as clusters. The goal is to uncover
interesting subgroups in the dataset.
What is the formula for AIC?
ANSWER AIC = -2*l + 2*p
l = maximized loglikelihood of the model on the training set
p = number of parameters
What is the formula for BIC?
ANSWER BIC = -2*l + p*ln(n)
l = maximized loglikelihood of the model on the training set
n = number of observations in the training set
Describe the pros and cons of using AIC in stepwise selection.
ANSWER The AIC penalty parameter is lower than that of BIC, so the AIC is less stringent than
BIC and leads to a more complex model. AIC is normally combined with backwards stepwise
selection.
Describe the pros and cons of using BIC in stepwise selection.
ANSWER The BIC penalty parameter is higher than that of AIC, so the BIC is more stringent
than AIC and leads to a less complex model. BIC is normally combined with forwards stepwise
selection.
Describe the pros and cons of using forward stepwise selection.
ANSWER Forward stepwise selection involves starting with the null model (only the intercept)
and adding one variable at a time until we no longer want to add anymore. We add the variable
that leads to the greatest improvement in the model, according to a pre-specified performance
metric.
It tends to lead to a much smaller (less complex) and interpretable model. However, it can be
underfitted in some regards.
Describe the pros and cons of using backwards stepwise selection.
ANSWER Backwards stepwise selection involves starting with the full model (all variables in
the dataset are accounted for), and dropping one variable at a time, depending on which
variable being dropped leads to the greatest improvement of model performance, according to
a pre-specified performance metric.
It tends to lead to a more complex model than forward selection, which can sometimes be
overfitted still.
, Discuss the pros and cons of using a random forest.
ANSWER Pros: Relative to a single tree, random forests are usually much more robust.
Combining all B of the base trees contributed substantial variance reduction, which results in
much more precise predictions.
Cons: Random forests are generally not as interpretable as base trees, and it takes a
considerably longer time to implement a random forest compared to a decision tree due to the
computational burden that comes with it
Describe the pros and cons of decision trees.
ANSWER Pros: It is easy to interpret and explain to non-technical audiences. It can also be
displayed graphically. It does a good job of modeling non-linear relationships and
automatically recognizes interactions. Categorical predictors are automatically handled
without binarization and variables are automatically selected.
Cons: Decision trees are more prone to overfitting than GLMs, even with pruning. In order to
capture the effects of a numeric variable, we need to split based on this variable repeatedly,
which gives rise to a complex tree with large depth. There is also a lack of model diagnostic
tools.
Describe the pros and cons of boosted trees.
ANSWER Pros: Boosted trees often perform better in terms of prediction accuracy than
random forests, due to their emphasis on bias reduction.
Cons: Boosted trees are more prone to overfitting than random forests. There is also a loss of
interpretability when compared to base trees.
Describe random forests.
ANSWER Random forests entail generating multiple bootstrapped (with replacement)
samples of the training set and fitting base trees in parallel, independently on each of the
samples. We combine all of these base trees to form an overall prediction. In each sample, a
random sample of m predictors is chosen as the split candidates of the p available features.
Describe boosted trees.
ANSWER Boosting builds a sequence of interdependent trees using information from
previously grown trees. In each iteration, we fit a tree to the residuals of the preceding tree,
and a scaled down version of the current tree's predictions is subtracted from the preceding
tree's residuals to form the new residuals. The whole process is repeated, with the effect being
that each tree will focus on predicting observations that the previous tree predicted poorly.
How can we tell how many PCs to use in PCA?
ANSWER We can use a scree plot, which is a plot of the principle component and the
proportion of variance explained. We choose the number of PCs by eyeballing the plot too look
Describe the Bias-Variance trade off.
ANSWER The bias-variance trade off relates to making your model a good balance of being
fitted to the training data and being predictive of the test data. When a model is fitted too well
(too complex), then it tends to lack predictive power. When a model predicts too well, it tends
to not be fitted too well to the training data.
Bias refers to the ability to capture the signal in the data. It is the part of the test error caused
by a model that is not flexible enough to capture the underlying signal.
Variance refers to the ability to capture the predictive performance. We lower variance by
making the model a better fit to future predictions.
Describe stepwise selection.
ANSWER Stepwise selection is a process of feature selection where we either start with a null
model (forward selection) or start with the full model (backwards selection) and add or remove
one variable at a time, depending on which variable causes the best improvement in the model
by adding or dropping it.
Define regularization.
ANSWER Regularization is a method for reducing the complexity of a linear model. We
consider a model that hosts all of the possible features, and we shrink the coefficient estimates
towards zero for those features that have limited predictive power. This is an alternative to
completely removing the features.
Define interaction.
ANSWER Interaction arises when the expected effect of one predictor on the target variable
depends on the value (or level) of another predictor.
Describe what pruning does.
ANSWER Pruning is a method of feature selection in decision trees. It involves removing
branches of the tree that lack in predictive performance, subject to a control parameter.
Describe PCA.
ANSWER Principal component analysis is a data analytic technique that transforms high
dimensional datasets into a smaller, more manageable set of representative variables that
capture most of the information in the original dataset. It combines multiple variables into new
features called Principal Components (PCs) which capture multiple variables into one.
Describe cluster analysis.
, ANSWER Cluster analysis is a data analytic tool for high dimensional datasets. It works by
partitioning observations into a set of distinct groups, known as clusters. The goal is to uncover
interesting subgroups in the dataset.
What is the formula for AIC?
ANSWER AIC = -2*l + 2*p
l = maximized loglikelihood of the model on the training set
p = number of parameters
What is the formula for BIC?
ANSWER BIC = -2*l + p*ln(n)
l = maximized loglikelihood of the model on the training set
n = number of observations in the training set
Describe the pros and cons of using AIC in stepwise selection.
ANSWER The AIC penalty parameter is lower than that of BIC, so the AIC is less stringent than
BIC and leads to a more complex model. AIC is normally combined with backwards stepwise
selection.
Describe the pros and cons of using BIC in stepwise selection.
ANSWER The BIC penalty parameter is higher than that of AIC, so the BIC is more stringent
than AIC and leads to a less complex model. BIC is normally combined with forwards stepwise
selection.
Describe the pros and cons of using forward stepwise selection.
ANSWER Forward stepwise selection involves starting with the null model (only the intercept)
and adding one variable at a time until we no longer want to add anymore. We add the variable
that leads to the greatest improvement in the model, according to a pre-specified performance
metric.
It tends to lead to a much smaller (less complex) and interpretable model. However, it can be
underfitted in some regards.
Describe the pros and cons of using backwards stepwise selection.
ANSWER Backwards stepwise selection involves starting with the full model (all variables in
the dataset are accounted for), and dropping one variable at a time, depending on which
variable being dropped leads to the greatest improvement of model performance, according to
a pre-specified performance metric.
It tends to lead to a more complex model than forward selection, which can sometimes be
overfitted still.
, Discuss the pros and cons of using a random forest.
ANSWER Pros: Relative to a single tree, random forests are usually much more robust.
Combining all B of the base trees contributed substantial variance reduction, which results in
much more precise predictions.
Cons: Random forests are generally not as interpretable as base trees, and it takes a
considerably longer time to implement a random forest compared to a decision tree due to the
computational burden that comes with it
Describe the pros and cons of decision trees.
ANSWER Pros: It is easy to interpret and explain to non-technical audiences. It can also be
displayed graphically. It does a good job of modeling non-linear relationships and
automatically recognizes interactions. Categorical predictors are automatically handled
without binarization and variables are automatically selected.
Cons: Decision trees are more prone to overfitting than GLMs, even with pruning. In order to
capture the effects of a numeric variable, we need to split based on this variable repeatedly,
which gives rise to a complex tree with large depth. There is also a lack of model diagnostic
tools.
Describe the pros and cons of boosted trees.
ANSWER Pros: Boosted trees often perform better in terms of prediction accuracy than
random forests, due to their emphasis on bias reduction.
Cons: Boosted trees are more prone to overfitting than random forests. There is also a loss of
interpretability when compared to base trees.
Describe random forests.
ANSWER Random forests entail generating multiple bootstrapped (with replacement)
samples of the training set and fitting base trees in parallel, independently on each of the
samples. We combine all of these base trees to form an overall prediction. In each sample, a
random sample of m predictors is chosen as the split candidates of the p available features.
Describe boosted trees.
ANSWER Boosting builds a sequence of interdependent trees using information from
previously grown trees. In each iteration, we fit a tree to the residuals of the preceding tree,
and a scaled down version of the current tree's predictions is subtracted from the preceding
tree's residuals to form the new residuals. The whole process is repeated, with the effect being
that each tree will focus on predicting observations that the previous tree predicted poorly.
How can we tell how many PCs to use in PCA?
ANSWER We can use a scree plot, which is a plot of the principle component and the
proportion of variance explained. We choose the number of PCs by eyeballing the plot too look
