ISYE 6501 Analytics Modeling Midterm 2
Question and Answer 2026 | Verified
• Greedy Variable Selection Methods -✓✓Forward Selection
Backward Elimination
Stepwise Regression
• Greedy Variable Selection - Definition -✓✓Step-by step model evaluation methods to
help find the best combination of variables for predictive models.
• Greedy Variable Selection - Pros -✓✓Fast
Good for initial exploration
• Greedy Variable Selection - Cons -✓✓May overfit to random patterns
May find locally good but globablly suboptimal solutions
Often overfit more than global methods
• Forward Selection -✓✓Greedy
Build up from nothing, adds the best new factor at each step, stops when no factor
improves the model enough
• Backward Elimination -✓✓Greedy
Narrow down from all variables, removes the worst factor at each step, stops when no
factor is bad enough to remove
• Stepwise Regression -✓✓Greedy
Bidirectional, combines forward and backward: after each step, immediately re-check
and remove any factors that are no longer useful
• Global Variable Selection Methods -✓✓LASSO Regression
Ridge Regression
Elastic Net
• Global Variable Selection - Definition -✓✓Methods that optimize over all factors
simultaneously rather than step-by step.
• Lasso Regression -✓✓Least Absolute Shrinkage and Selection Operator
Variable Selection
Adds constraint to linear regression that limits the total sum of absolute values of
coefficients (high variance)
Constraint region shape = diamond
• Lasso Formula -✓✓Minimize SSE subject to sum(|aj|) <=T
, • Ridge Regression -✓✓Limits the sum of squares by shrinking all coefficients towards
0, reduces overfitting by reducing model variance (high bias)
Constraint Region shape = circle
• Ridge Regression Formula -✓✓minimize SSE subject to sum(aj^2) <=T
• Elastic Net -✓✓Combines Lasso and ridge regressions. Useful when both benefits are
needed, but also inherits drawbacks of each
• Elastic Net Formula -✓✓Sum(|aj|) + sum(aj^2) <T
• Bias -✓✓Error from missing real patterns (underfitting)
More bias = predictions cluster toward constant regardless of input
• Variance -✓✓Error from fitting random noise (overfitting)
More bias = predictions change dramatically with small data changes
• Bias-Variance Tradeoff -✓✓Fundamental tradeoff between fitting real patterns and
fitting random noise. Point in the middle where total prediction error (bias + variance) is
minimized.
• Choosing Variable Selection Method - Initial Exploratory Analysis -✓✓Greedy models
Forward/backwards/stepwise regressions
Fast, gives a starting point
• Choosing Variable Selection Method - Building a Production Model -✓✓LASSO or
Elastic Net
Better predictive accuracy
• Choosing Variable Selection Method - Highly Correlated Predictors -✓✓Check
correlations manually
Elastic Net or Ridge may help
• Choosing Variable Selection Method - Need Interpretable Model -✓✓LASSO (fewer
variables)
Check practical effect sizes
• Choosing Variable Selection Method - Model Appears Overfit -✓✓Ridge Regression to
shrink coefficients
Reduce Tau Ridge carefully
• Design of Experiments -✓✓Designing a process for collecting the most useful subset
of data quickly and efficiently
Question and Answer 2026 | Verified
• Greedy Variable Selection Methods -✓✓Forward Selection
Backward Elimination
Stepwise Regression
• Greedy Variable Selection - Definition -✓✓Step-by step model evaluation methods to
help find the best combination of variables for predictive models.
• Greedy Variable Selection - Pros -✓✓Fast
Good for initial exploration
• Greedy Variable Selection - Cons -✓✓May overfit to random patterns
May find locally good but globablly suboptimal solutions
Often overfit more than global methods
• Forward Selection -✓✓Greedy
Build up from nothing, adds the best new factor at each step, stops when no factor
improves the model enough
• Backward Elimination -✓✓Greedy
Narrow down from all variables, removes the worst factor at each step, stops when no
factor is bad enough to remove
• Stepwise Regression -✓✓Greedy
Bidirectional, combines forward and backward: after each step, immediately re-check
and remove any factors that are no longer useful
• Global Variable Selection Methods -✓✓LASSO Regression
Ridge Regression
Elastic Net
• Global Variable Selection - Definition -✓✓Methods that optimize over all factors
simultaneously rather than step-by step.
• Lasso Regression -✓✓Least Absolute Shrinkage and Selection Operator
Variable Selection
Adds constraint to linear regression that limits the total sum of absolute values of
coefficients (high variance)
Constraint region shape = diamond
• Lasso Formula -✓✓Minimize SSE subject to sum(|aj|) <=T
, • Ridge Regression -✓✓Limits the sum of squares by shrinking all coefficients towards
0, reduces overfitting by reducing model variance (high bias)
Constraint Region shape = circle
• Ridge Regression Formula -✓✓minimize SSE subject to sum(aj^2) <=T
• Elastic Net -✓✓Combines Lasso and ridge regressions. Useful when both benefits are
needed, but also inherits drawbacks of each
• Elastic Net Formula -✓✓Sum(|aj|) + sum(aj^2) <T
• Bias -✓✓Error from missing real patterns (underfitting)
More bias = predictions cluster toward constant regardless of input
• Variance -✓✓Error from fitting random noise (overfitting)
More bias = predictions change dramatically with small data changes
• Bias-Variance Tradeoff -✓✓Fundamental tradeoff between fitting real patterns and
fitting random noise. Point in the middle where total prediction error (bias + variance) is
minimized.
• Choosing Variable Selection Method - Initial Exploratory Analysis -✓✓Greedy models
Forward/backwards/stepwise regressions
Fast, gives a starting point
• Choosing Variable Selection Method - Building a Production Model -✓✓LASSO or
Elastic Net
Better predictive accuracy
• Choosing Variable Selection Method - Highly Correlated Predictors -✓✓Check
correlations manually
Elastic Net or Ridge may help
• Choosing Variable Selection Method - Need Interpretable Model -✓✓LASSO (fewer
variables)
Check practical effect sizes
• Choosing Variable Selection Method - Model Appears Overfit -✓✓Ridge Regression to
shrink coefficients
Reduce Tau Ridge carefully
• Design of Experiments -✓✓Designing a process for collecting the most useful subset
of data quickly and efficiently