ISYE 6501 MIDTERM 2 COMPREHENSIVE
QUESTIONS AND COMPLETE SOLUTIONS
◉ Why are simple models better?
Answer: Less data is required, less chance of insignificant factors,
easier to interpret
◉ Examples of factors that are illegal to use
Answer: race, sex, religion, marital status, or any factors that are
highly correlated with forbidden ones
◉ What is forward selection?
Answer: Start with a model with no factors, at each step, find each
best new factor to add to model, and put it in if good enough
(parameter of your choice like p value<=0.15) improvement; when
there is no factor that is good enough or if we add enough factors,
we stop; can remove any factors at the end
◉ What is backward selection?
Answer: Start with all factors, and at each step, we find worst factor,
and remove from model and we keep going until no factor bad
enough to remove or we reach the number of factors we want
,◉ What is stepwise regression?
Answer: it is a combination of forward selection and backward
elimination. We can either start with all factors or no factors and at
each step we remove or add a factor. As we go through the procedure
after adding each new factor and at the end we eliminate right away
factors that no longer appear; Allows model to adjust if factor we
earlier thought we needed no longer seems necessary thanks to new
factors added
◉ What is a greedy algorithm?
Answer: At each step, it does the one thing that looks best without
future options are considered
◉ What are potential metrics to determine whether to add or
remove factors in variable selection?
Answer: p value, AIC, BIC, R^2
◉ What are methods that are based on optimization models that
make decisions globally looking at all options at the same time when
it comes to variable selection?
Answer: LASSO, Elastic Net, Ridge Regression (not variable
selection)
◉ LASSO Approach
, Answer: Adds constraint to standard regression equation: we want
to minimize the sum of squared errors, but also the sum of
coefficients cannot be too large; add threshold t - budget to use on
coefficients; need to scale data - uses absolute value of coefficients
◉ Factors to consider when picking the right value of t for LASSO
approach?
Answer: Number of variables and quality of model
◉ Elastic Net
Answer: Constrain combination of absolute value of coefficients and
their squares, need to scale data, choose t and lambda values
◉ Ridge Regression
Answer: Constrain the regression equation by the coefficient
squares - doesn't do variable selection
◉ LASSO vs Ridge
Answer: The equation to choose coefficients that minimize the total
error is the same; however, the restrictions to solutions are
different; LASSO is based on absolute sum of coefficients and Ridge
is based on sum of squares of coefficients
QUESTIONS AND COMPLETE SOLUTIONS
◉ Why are simple models better?
Answer: Less data is required, less chance of insignificant factors,
easier to interpret
◉ Examples of factors that are illegal to use
Answer: race, sex, religion, marital status, or any factors that are
highly correlated with forbidden ones
◉ What is forward selection?
Answer: Start with a model with no factors, at each step, find each
best new factor to add to model, and put it in if good enough
(parameter of your choice like p value<=0.15) improvement; when
there is no factor that is good enough or if we add enough factors,
we stop; can remove any factors at the end
◉ What is backward selection?
Answer: Start with all factors, and at each step, we find worst factor,
and remove from model and we keep going until no factor bad
enough to remove or we reach the number of factors we want
,◉ What is stepwise regression?
Answer: it is a combination of forward selection and backward
elimination. We can either start with all factors or no factors and at
each step we remove or add a factor. As we go through the procedure
after adding each new factor and at the end we eliminate right away
factors that no longer appear; Allows model to adjust if factor we
earlier thought we needed no longer seems necessary thanks to new
factors added
◉ What is a greedy algorithm?
Answer: At each step, it does the one thing that looks best without
future options are considered
◉ What are potential metrics to determine whether to add or
remove factors in variable selection?
Answer: p value, AIC, BIC, R^2
◉ What are methods that are based on optimization models that
make decisions globally looking at all options at the same time when
it comes to variable selection?
Answer: LASSO, Elastic Net, Ridge Regression (not variable
selection)
◉ LASSO Approach
, Answer: Adds constraint to standard regression equation: we want
to minimize the sum of squared errors, but also the sum of
coefficients cannot be too large; add threshold t - budget to use on
coefficients; need to scale data - uses absolute value of coefficients
◉ Factors to consider when picking the right value of t for LASSO
approach?
Answer: Number of variables and quality of model
◉ Elastic Net
Answer: Constrain combination of absolute value of coefficients and
their squares, need to scale data, choose t and lambda values
◉ Ridge Regression
Answer: Constrain the regression equation by the coefficient
squares - doesn't do variable selection
◉ LASSO vs Ridge
Answer: The equation to choose coefficients that minimize the total
error is the same; however, the restrictions to solutions are
different; LASSO is based on absolute sum of coefficients and Ridge
is based on sum of squares of coefficients
