ISYE 6501 FINAL EXAM REVIEW COMPLETE
QUESTIONS AND SOLUTIONS
◉ Greedy Variable Selection Methods.
Answer: Forward Selection
Backward Elimination
Stepwise Regression
◉ Greedy Variable Selection - Definition.
Answer: Step-by step model evaluation methods to help find the best
combination of variables for predictive models.
◉ Greedy Variable Selection - Pros.
Answer: Fast
Good for initial exploration
◉ Greedy Variable Selection - Cons.
Answer: May overfit to random patterns
May find locally good but globablly suboptimal solutions
Often overfit more than global methods
,◉ Forward Selection.
Answer: Greedy
Build up from nothing, adds the best new factor at each step, stops
when no factor improves the model enough
◉ Backward Elimination.
Answer: Greedy
Narrow down from all variables, removes the worst factor at each
step, stops when no factor is bad enough to remove
◉ Stepwise Regression.
Answer: Greedy
Bidirectional, combines forward and backward: after each step,
immediately re-check and remove any factors that are no longer
useful
◉ Global Variable Selection Methods.
Answer: LASSO Regression
Ridge Regression
Elastic Net
◉ Global Variable Selection - Definition.
,Answer: Methods that optimize over all factors simultaneously
rather than step-by step.
◉ Lasso Regression.
Answer: Least Absolute Shrinkage and Selection Operator
Variable Selection
Adds constraint to linear regression that limits the total sum of
absolute values of coefficients (high variance)
Constraint region shape = diamond
◉ Lasso Formula.
Answer: Minimize SSE subject to sum(|aj|) <=T
◉ Ridge Regression.
Answer: Limits the sum of squares by shrinking all coefficients
towards 0, reduces overfitting by reducing model variance (high
bias)
Constraint Region shape = circle
◉ Ridge Regression Formula.
Answer: minimize SSE subject to sum(aj^2) <=T
◉ Elastic Net.
, Answer: Combines Lasso and ridge regressions. Useful when both
benefits are needed, but also inherits drawbacks of each
◉ Elastic Net Formula.
Answer: Sum(|aj|) + sum(aj^2) <T
◉ Bias.
Answer: Error from missing real patterns (underfitting)
More bias = predictions cluster toward constant regardless of input
◉ Variance.
Answer: Error from fitting random noise (overfitting)
More bias = predictions change dramatically with small data
changes
◉ Bias-Variance Tradeoff.
Answer: Fundamental tradeoff between fitting real patterns and
fitting random noise. Point in the middle where total prediction
error (bias + variance) is minimized.
◉ Choosing Variable Selection Method - Initial Exploratory Analysis.
Answer: Greedy models
Forward/backwards/stepwise regressions
QUESTIONS AND SOLUTIONS
◉ Greedy Variable Selection Methods.
Answer: Forward Selection
Backward Elimination
Stepwise Regression
◉ Greedy Variable Selection - Definition.
Answer: Step-by step model evaluation methods to help find the best
combination of variables for predictive models.
◉ Greedy Variable Selection - Pros.
Answer: Fast
Good for initial exploration
◉ Greedy Variable Selection - Cons.
Answer: May overfit to random patterns
May find locally good but globablly suboptimal solutions
Often overfit more than global methods
,◉ Forward Selection.
Answer: Greedy
Build up from nothing, adds the best new factor at each step, stops
when no factor improves the model enough
◉ Backward Elimination.
Answer: Greedy
Narrow down from all variables, removes the worst factor at each
step, stops when no factor is bad enough to remove
◉ Stepwise Regression.
Answer: Greedy
Bidirectional, combines forward and backward: after each step,
immediately re-check and remove any factors that are no longer
useful
◉ Global Variable Selection Methods.
Answer: LASSO Regression
Ridge Regression
Elastic Net
◉ Global Variable Selection - Definition.
,Answer: Methods that optimize over all factors simultaneously
rather than step-by step.
◉ Lasso Regression.
Answer: Least Absolute Shrinkage and Selection Operator
Variable Selection
Adds constraint to linear regression that limits the total sum of
absolute values of coefficients (high variance)
Constraint region shape = diamond
◉ Lasso Formula.
Answer: Minimize SSE subject to sum(|aj|) <=T
◉ Ridge Regression.
Answer: Limits the sum of squares by shrinking all coefficients
towards 0, reduces overfitting by reducing model variance (high
bias)
Constraint Region shape = circle
◉ Ridge Regression Formula.
Answer: minimize SSE subject to sum(aj^2) <=T
◉ Elastic Net.
, Answer: Combines Lasso and ridge regressions. Useful when both
benefits are needed, but also inherits drawbacks of each
◉ Elastic Net Formula.
Answer: Sum(|aj|) + sum(aj^2) <T
◉ Bias.
Answer: Error from missing real patterns (underfitting)
More bias = predictions cluster toward constant regardless of input
◉ Variance.
Answer: Error from fitting random noise (overfitting)
More bias = predictions change dramatically with small data
changes
◉ Bias-Variance Tradeoff.
Answer: Fundamental tradeoff between fitting real patterns and
fitting random noise. Point in the middle where total prediction
error (bias + variance) is minimized.
◉ Choosing Variable Selection Method - Initial Exploratory Analysis.
Answer: Greedy models
Forward/backwards/stepwise regressions
