ISYE 6501 Analytics Modeling Midterm 2
Question and Answer | A+
Main reasons to limit number of factors in model -✓✓Overfitting - when # of factors is
close to or larger than # of data points; Simplicity - simple models are better; reduce the
number of correlated variables; certain variables might be hard to collect data or
expensive; some variables are missing data or hard to use
Why are simple models better? -✓✓Less data is required, less chance of insignificant
factors, easier to interpret
Examples of factors that are illegal to use -✓✓race, sex, religion, marital status, or any
factors that are highly correlated with forbidden ones
What is forward selection? -✓✓Start with a model with no factors, at each step, find
each best new factor to add to model, and put it in if good enough (parameter of your
choice like p value<=0.15) improvement; when there is no factor that is good enough or
if we add enough factors, we stop; can remove any factors at the end
What is backward selection? -✓✓Start with all factors, and at each step, we find worst
factor, and remove from model and we keep going until no factor bad enough to remove
or we reach the number of factors we want
What is stepwise regression? -✓✓it is a combination of forward selection and backward
elimination. We can either start with all factors or no factors and at each step we
remove or add a factor. As we go through the procedure after adding each new factor
and at the end we eliminate right away factors that no longer appear; Allows model to
adjust if factor we earlier thought we needed no longer seems necessary thanks to new
factors added
What is a greedy algorithm? -✓✓At each step, it does the one thing that looks best
without future options are considered
What are potential metrics to determine whether to add or remove factors in variable
selection? -✓✓p value, AIC, BIC, R^2
What are methods that are based on optimization models that make decisions globally
looking at all options at the same time when it comes to variable selection? -✓✓LASSO,
Elastic Net, Ridge Regression (not variable selection)
LASSO Approach -✓✓Adds constraint to standard regression equation: we want to
minimize the sum of squared errors, but also the sum of coefficients cannot be too
, large; add threshold t - budget to use on coefficients; need to scale data - uses absolute
value of coefficients
Factors to consider when picking the right value of t for LASSO approach? -✓✓Number
of variables and quality of model
Elastic Net -✓✓Constrain combination of absolute value of coefficients and their
squares, need to scale data, choose t and lambda values
Ridge Regression -✓✓Constrain the regression equation by the coefficient squares -
doesn't do variable selection
LASSO vs Ridge -✓✓The equation to choose coefficients that minimize the total error is
the same; however, the restrictions to solutions are different; LASSO is based on
absolute sum of coefficients and Ridge is based on sum of squares of coefficients
In Lasso, why are variables usually removed and if not, how to adjust to make sure of
removal? -✓✓Based on restriction equation and error equation, there are points where
the error function will reach the threshold where certain coefficients are 0, but the more
predictors we have and the more restrictive tau is, the higher the chances of 0; if not,
reduce tau
Two kinds of pattern in data -✓✓real patterns; random patterns - look like real effects,
but are random effects; models just see patterns, cannot tell if real or random
Bias-variance tradeoff -✓✓· Underfit model has high bias and low variance - underfitting
the real effects while eliminating variance from random effect
· Overfit models has low bias and high variance - good fits to real patterns, getting lot of
unwanted variance from random patterns
Forward/Backward/Step vs LASSO/Elastic -✓✓The first is good for initial analysis,
points out which variables are important to investigate further, and do it quickly, but can
get variables that fit more to random effects but appear good due to better fit, but when
you test it, it does not do well
The latter computes slower, but give better predictive models
Advantages of Elastic Net -✓✓Variable selection of LASSO, predictive benefits of Ridge
Regression
Disadvantages of Elastic Net -✓✓Arbitrarily rules out some correlated variables like
LASSO - example if two variables are highly correlated, LASSO will pick one, even
though the other might be better (Ridge will pick both and coefficients will shrink and get
closer to each other)
Question and Answer | A+
Main reasons to limit number of factors in model -✓✓Overfitting - when # of factors is
close to or larger than # of data points; Simplicity - simple models are better; reduce the
number of correlated variables; certain variables might be hard to collect data or
expensive; some variables are missing data or hard to use
Why are simple models better? -✓✓Less data is required, less chance of insignificant
factors, easier to interpret
Examples of factors that are illegal to use -✓✓race, sex, religion, marital status, or any
factors that are highly correlated with forbidden ones
What is forward selection? -✓✓Start with a model with no factors, at each step, find
each best new factor to add to model, and put it in if good enough (parameter of your
choice like p value<=0.15) improvement; when there is no factor that is good enough or
if we add enough factors, we stop; can remove any factors at the end
What is backward selection? -✓✓Start with all factors, and at each step, we find worst
factor, and remove from model and we keep going until no factor bad enough to remove
or we reach the number of factors we want
What is stepwise regression? -✓✓it is a combination of forward selection and backward
elimination. We can either start with all factors or no factors and at each step we
remove or add a factor. As we go through the procedure after adding each new factor
and at the end we eliminate right away factors that no longer appear; Allows model to
adjust if factor we earlier thought we needed no longer seems necessary thanks to new
factors added
What is a greedy algorithm? -✓✓At each step, it does the one thing that looks best
without future options are considered
What are potential metrics to determine whether to add or remove factors in variable
selection? -✓✓p value, AIC, BIC, R^2
What are methods that are based on optimization models that make decisions globally
looking at all options at the same time when it comes to variable selection? -✓✓LASSO,
Elastic Net, Ridge Regression (not variable selection)
LASSO Approach -✓✓Adds constraint to standard regression equation: we want to
minimize the sum of squared errors, but also the sum of coefficients cannot be too
, large; add threshold t - budget to use on coefficients; need to scale data - uses absolute
value of coefficients
Factors to consider when picking the right value of t for LASSO approach? -✓✓Number
of variables and quality of model
Elastic Net -✓✓Constrain combination of absolute value of coefficients and their
squares, need to scale data, choose t and lambda values
Ridge Regression -✓✓Constrain the regression equation by the coefficient squares -
doesn't do variable selection
LASSO vs Ridge -✓✓The equation to choose coefficients that minimize the total error is
the same; however, the restrictions to solutions are different; LASSO is based on
absolute sum of coefficients and Ridge is based on sum of squares of coefficients
In Lasso, why are variables usually removed and if not, how to adjust to make sure of
removal? -✓✓Based on restriction equation and error equation, there are points where
the error function will reach the threshold where certain coefficients are 0, but the more
predictors we have and the more restrictive tau is, the higher the chances of 0; if not,
reduce tau
Two kinds of pattern in data -✓✓real patterns; random patterns - look like real effects,
but are random effects; models just see patterns, cannot tell if real or random
Bias-variance tradeoff -✓✓· Underfit model has high bias and low variance - underfitting
the real effects while eliminating variance from random effect
· Overfit models has low bias and high variance - good fits to real patterns, getting lot of
unwanted variance from random patterns
Forward/Backward/Step vs LASSO/Elastic -✓✓The first is good for initial analysis,
points out which variables are important to investigate further, and do it quickly, but can
get variables that fit more to random effects but appear good due to better fit, but when
you test it, it does not do well
The latter computes slower, but give better predictive models
Advantages of Elastic Net -✓✓Variable selection of LASSO, predictive benefits of Ridge
Regression
Disadvantages of Elastic Net -✓✓Arbitrarily rules out some correlated variables like
LASSO - example if two variables are highly correlated, LASSO will pick one, even
though the other might be better (Ridge will pick both and coefficients will shrink and get
closer to each other)