ISYE 6501 FINAL PAPER REVIEW SHEET
UPDATED QUESTIONS AND SOLUTIONS
●● Why are simple models better than complex ones
Answer: less data is required; less chance of insignificant factors and
easier to interpret
●● what is forward selection
Answer: we select the best new factor and see if it's good enough (R^2,
AIC, or p-value) add it to our model and fit the model with the current
set of factors. Then at the end we remove factors that are lower than a
certain threshold
●● what is backward elimination
Answer: we start with all factors and find the worst on a supplied
threshold (p = 0.15). If it is worse we remove it and start the process
over. We do that until we have the number of factors that we want and
then we move the factors lower than a second threshold (p = .05) and fit
the model with all set of factors
●● what is stepwise regression
Answer: it is a combination of forward selection and backward
elimination. We can either start with all factors or no factors and at each
step we remove or add a factor. As we go through the procedure after
,adding each new factor and at the end we eliminate right away factors
that no longer appear.
●● what type of algorithms are stepwise selection?
Answer: Greedy algorithms - at each step they take one thing that looks
best
●● what is LASSO
Answer: a variable selection method where the coefficients are
determined by both minimizing the squared error and the sum of their
absolute value not being over a certain threshold t
●● How do you choose t in LASSO
Answer: use the lasso approach with different values of t and see which
gives the best trade off
●● why do we have to scale the data for LASSO
Answer: if we don't, the measure of the data will artificially affect how
big the coefficients need to be
●● What is elastic net?
Answer: A variable selection method that works by minimizing the
squared error and constraining the combination of absolute values of
coefficients and their squares
,●● what is a key difference between stepwise regresson and lasso
regression ***
Answer: If the data is not scaled, the coefficients can have artificially
different orders of magnitude, which means they'll have unbalanced
effects on the lasso constraint.
●● Why doesn't Ridge Regression perform variable selection?
Answer: The coefficients values are squared so they go closer to zero or
regularizes them, but the coefficient values are never equal to zero
●● What are the pros and cons of Greedy Algorithms (Forward
selection, stepwise elimination, stepwise regression)
Answer: Good for initial analysis but often don't perform as well on
other data because they fit more to random effects than you'd like and
appear to have a better fit
●● What are the pros and cons of LASSO, Ridge and Elastic Net
Answer: They are slower but help make models that make better
predictions
●● Which two methods does elastic net look like it combines and what
are the downsides from it?
Answer: Ridge Regression and LASSO.
, Advantages: variable selection from LASSO and Predictive benefits of
Ridge.
Disadvantages: Arbitrarily rules out some correlated variables (e.g.
LASSO doesn't know which one should be left out); Underestimates
coefficients of very predictive variables (i.e. Ridge Regression)
●● What are some downsides of surveys?
Answer: Even if you have what appears to be a representative sample in
simple ways, maybe it isn't in more complex ways.
●● If we're testing to see whether red cars sell for higher prices than
blue cars, we need to account for the type and age of the cars in our data
set. This is called:
Answer: Controlling
●● what is a blocking factor ***
Answer: a source of variability that is not of primary interest to the
experimenter
●● what is an example of a blocking factor
Answer: The type of car, sports car or family car, is a blocking factor
that it could account for some of the difference between red cars and
UPDATED QUESTIONS AND SOLUTIONS
●● Why are simple models better than complex ones
Answer: less data is required; less chance of insignificant factors and
easier to interpret
●● what is forward selection
Answer: we select the best new factor and see if it's good enough (R^2,
AIC, or p-value) add it to our model and fit the model with the current
set of factors. Then at the end we remove factors that are lower than a
certain threshold
●● what is backward elimination
Answer: we start with all factors and find the worst on a supplied
threshold (p = 0.15). If it is worse we remove it and start the process
over. We do that until we have the number of factors that we want and
then we move the factors lower than a second threshold (p = .05) and fit
the model with all set of factors
●● what is stepwise regression
Answer: it is a combination of forward selection and backward
elimination. We can either start with all factors or no factors and at each
step we remove or add a factor. As we go through the procedure after
,adding each new factor and at the end we eliminate right away factors
that no longer appear.
●● what type of algorithms are stepwise selection?
Answer: Greedy algorithms - at each step they take one thing that looks
best
●● what is LASSO
Answer: a variable selection method where the coefficients are
determined by both minimizing the squared error and the sum of their
absolute value not being over a certain threshold t
●● How do you choose t in LASSO
Answer: use the lasso approach with different values of t and see which
gives the best trade off
●● why do we have to scale the data for LASSO
Answer: if we don't, the measure of the data will artificially affect how
big the coefficients need to be
●● What is elastic net?
Answer: A variable selection method that works by minimizing the
squared error and constraining the combination of absolute values of
coefficients and their squares
,●● what is a key difference between stepwise regresson and lasso
regression ***
Answer: If the data is not scaled, the coefficients can have artificially
different orders of magnitude, which means they'll have unbalanced
effects on the lasso constraint.
●● Why doesn't Ridge Regression perform variable selection?
Answer: The coefficients values are squared so they go closer to zero or
regularizes them, but the coefficient values are never equal to zero
●● What are the pros and cons of Greedy Algorithms (Forward
selection, stepwise elimination, stepwise regression)
Answer: Good for initial analysis but often don't perform as well on
other data because they fit more to random effects than you'd like and
appear to have a better fit
●● What are the pros and cons of LASSO, Ridge and Elastic Net
Answer: They are slower but help make models that make better
predictions
●● Which two methods does elastic net look like it combines and what
are the downsides from it?
Answer: Ridge Regression and LASSO.
, Advantages: variable selection from LASSO and Predictive benefits of
Ridge.
Disadvantages: Arbitrarily rules out some correlated variables (e.g.
LASSO doesn't know which one should be left out); Underestimates
coefficients of very predictive variables (i.e. Ridge Regression)
●● What are some downsides of surveys?
Answer: Even if you have what appears to be a representative sample in
simple ways, maybe it isn't in more complex ways.
●● If we're testing to see whether red cars sell for higher prices than
blue cars, we need to account for the type and age of the cars in our data
set. This is called:
Answer: Controlling
●● what is a blocking factor ***
Answer: a source of variability that is not of primary interest to the
experimenter
●● what is an example of a blocking factor
Answer: The type of car, sports car or family car, is a blocking factor
that it could account for some of the difference between red cars and
