ISYE 6501 EXAM UPDATED ACTUAL
QUESTIONS AND CORRECT ANSWERS
◉ Greedy Variable Selection Methods
Answer: Forward Selection
Backward Elimination
Stepwise Regression
◉ Greedy Variable Selection - Definition
Answer: Step-by step model evaluation methods to help find the
best combination of variables for predictive models.
◉ Greedy Variable Selection - Pros
Answer: Fast
Good for initial exploration
◉ Greedy Variable Selection - Cons
Answer: May overfit to random patterns
May find locally good but globablly suboptimal solutions
Often overfit more than global methods
◉ Forward Selection
,Answer: Greedy
Build up from nothing, adds the best new factor at each step, stops
when no factor improves the model enough
◉ Backward Elimination
Answer: Greedy
Narrow down from all variables, removes the worst factor at each
step, stops when no factor is bad enough to remove
◉ Stepwise Regression
Answer: Greedy
Bidirectional, combines forward and backward: after each step,
immediately re-check and remove any factors that are no longer
useful
◉ Global Variable Selection Methods
Answer: LASSO Regression
Ridge Regression
Elastic Net
◉ Global Variable Selection - Definition
Answer: Methods that optimize over all factors simultaneously
rather than step-by step.
◉ Lasso Regression
,Answer: Least Absolute Shrinkage and Selection Operator
Variable Selection
Adds constraint to linear regression that limits the total sum of
absolute values of coefficients (high variance)
Constraint region shape = diamond
◉ Lasso Formula
Answer: Minimize SSE subject to sum(|aj|) <=T
◉ Ridge Regression
Answer: Limits the sum of squares by shrinking all coefficients
towards 0, reduces overfitting by reducing model variance (high
bias)
Constraint Region shape = circle
◉ Ridge Regression Formula
Answer: minimize SSE subject to sum(aj^2) <=T
◉ Elastic Net
Answer: Combines Lasso and ridge regressions. Useful when both
benefits are needed, but also inherits drawbacks of each
◉ Elastic Net Formula
Answer: Sum(|aj|) + sum(aj^2) <T
, ◉ Bias
Answer: Error from missing real patterns (underfitting)
More bias = predictions cluster toward constant regardless of
input
◉ Variance
Answer: Error from fitting random noise (overfitting)
More bias = predictions change dramatically with small data
changes
◉ Bias-Variance Tradeoff
Answer: Fundamental tradeoff between fitting real patterns and
fitting random noise. Point in the middle where total prediction
error (bias + variance) is minimized.
◉ Choosing Variable Selection Method - Initial Exploratory
Analysis
Answer: Greedy models
Forward/backwards/stepwise regressions
Fast, gives a starting point
◉ Choosing Variable Selection Method - Building a Production
Model
Answer: LASSO or Elastic Net
QUESTIONS AND CORRECT ANSWERS
◉ Greedy Variable Selection Methods
Answer: Forward Selection
Backward Elimination
Stepwise Regression
◉ Greedy Variable Selection - Definition
Answer: Step-by step model evaluation methods to help find the
best combination of variables for predictive models.
◉ Greedy Variable Selection - Pros
Answer: Fast
Good for initial exploration
◉ Greedy Variable Selection - Cons
Answer: May overfit to random patterns
May find locally good but globablly suboptimal solutions
Often overfit more than global methods
◉ Forward Selection
,Answer: Greedy
Build up from nothing, adds the best new factor at each step, stops
when no factor improves the model enough
◉ Backward Elimination
Answer: Greedy
Narrow down from all variables, removes the worst factor at each
step, stops when no factor is bad enough to remove
◉ Stepwise Regression
Answer: Greedy
Bidirectional, combines forward and backward: after each step,
immediately re-check and remove any factors that are no longer
useful
◉ Global Variable Selection Methods
Answer: LASSO Regression
Ridge Regression
Elastic Net
◉ Global Variable Selection - Definition
Answer: Methods that optimize over all factors simultaneously
rather than step-by step.
◉ Lasso Regression
,Answer: Least Absolute Shrinkage and Selection Operator
Variable Selection
Adds constraint to linear regression that limits the total sum of
absolute values of coefficients (high variance)
Constraint region shape = diamond
◉ Lasso Formula
Answer: Minimize SSE subject to sum(|aj|) <=T
◉ Ridge Regression
Answer: Limits the sum of squares by shrinking all coefficients
towards 0, reduces overfitting by reducing model variance (high
bias)
Constraint Region shape = circle
◉ Ridge Regression Formula
Answer: minimize SSE subject to sum(aj^2) <=T
◉ Elastic Net
Answer: Combines Lasso and ridge regressions. Useful when both
benefits are needed, but also inherits drawbacks of each
◉ Elastic Net Formula
Answer: Sum(|aj|) + sum(aj^2) <T
, ◉ Bias
Answer: Error from missing real patterns (underfitting)
More bias = predictions cluster toward constant regardless of
input
◉ Variance
Answer: Error from fitting random noise (overfitting)
More bias = predictions change dramatically with small data
changes
◉ Bias-Variance Tradeoff
Answer: Fundamental tradeoff between fitting real patterns and
fitting random noise. Point in the middle where total prediction
error (bias + variance) is minimized.
◉ Choosing Variable Selection Method - Initial Exploratory
Analysis
Answer: Greedy models
Forward/backwards/stepwise regressions
Fast, gives a starting point
◉ Choosing Variable Selection Method - Building a Production
Model
Answer: LASSO or Elastic Net
