ISYE-6501: Homework 8
Question 11.1
Using the crime data set uscrime.txt from Questions 8.2, 9.1, and 10.1, build a regression
model using:
1. Stepwise regression
2. Lasso
3. Elastic net
For Parts 2 and 3, remember to scale the data first – otherwise, the regression coefficients
will be on different scales and the constraint won’t have the desired effect. For Parts 2 and 3,
use the glmnet function in R.
TL;DR R^2 Values for the following models:
Plain, ole Linear Regression: 0.803
Stepwise: 0.788
Stepwise CV: 0.585
LASSO: 0.797
LASSO CV: 0.505
EN (0.25): 0.798
EN (0.25) CV: 0.544
Stepwise Regression:
The step function was used with a forward and backward model for the lower and
upper formulas. The step found with the lowest AIC is below, resulting in the formula:
Crime = -6426.10102 + 93.32155* M + 180.12011*Ed + 102.65316*Po1 + 22.33975*M.F
- 6086.63315* U1 + 187.34512*U2 + 61.33494*Ineq - 3796.03183*Prob
Stepwise regression found these variables to be insignificant to reducing AIC: Wealth, Pop,
Po2, So, LF, NW, or Time.
Step: AIC=503.93
Crime ~ M + Ed + Po1 + M.F + U1 + U2 + Ineq + Prob
Df Sum of Sq RSS AIC
<none> 1453068 503.93
+ Wealth 1 26493 1426575 505.07
- M.F 1 103159 1556227 505.16
+ Pop 1 16697 1436371 505.39
+ Po2 1 14148 1438919 505.47
+ So 1 9329 1443739 505.63
,+ LF 1 4374 1448694 505.79
+ NW 1 3799 1449269 505.81
+ Time 1 2293 1450775 505.86
- U1 1 127044 1580112 505.87
- Prob 1 247978 1701046 509.34
- U2 1 255443 1708511 509.55
- M 1 296790 1749858 510.67
- Ed 1 445788 1898855 514.51
- Ineq 1 738244 2191312 521.24
- Po1 1 1672038 3125105 537.93
> stepwise_step1
Call:
lm(formula = Crime ~ M + Ed + Po1 + M.F + U1 + U2 + Ineq +
Prob, data = crime)
Coefficients:
(Intercept) M Ed Po1 M.F U1
U2
-6426.10 93.32 180.12 102.65 22.34 -6086.63
187.35
Ineq Prob
61.33 -3796.03
The p-values were less than 0.05 for the intercept, M, Ed, Po1, U2, Ineq and Prob, indicating
statistical significance of these factors. MF and U1 were the only ones that did not.
Call:
lm(formula = Crime ~ M + Ed + Po1 + M.F + U1 + U2 + Ineq + Prob,
data = crime)
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -6426.10 1194.61 -5.379 4.04e-06 ***
M 93.32 33.50 2.786 0.00828 **
Ed 180.12 52.75 3.414 0.00153 **
Po1 102.65 15.52 6.613 8.26e-08 ***
M.F 22.34 13.60 1.642 0.10874
U1 -6086.63 3339.27 -1.823 0.07622 .
U2 187.35 72.48 2.585 0.01371 *
Ineq 61.33 13.96 4.394 8.63e-05 ***
Prob -3796.03 1490.65 -2.547 0.01505 *
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’0.1‘’1
These results were then compared to the original, full regression model and a 5 -fold cross
validated model’s R^2 values. The original model was 0.803. The stepwise model was 0.788.
,The cross-validated model was 0.585. The cross validated predicted values per fold are
shown below, along with the code. Full code and results-in-line are in the appendix.
Unset
rm(list = ls())
set.seed(5)
crime <- read.table("~/DocumentsLocal/GT/ISYE-
6501/hw7/uscrime.txt", stringsAsFactors = FALSE, header = TRUE)
# Backward elimination
back_model <- lm(Crime~., data = crime)
# Forward selection
forward_model <- lm(Crime~1, data = crime)
# Stepwise regression
stepwise_model <- lm(Crime~., data = crime)
stepwise_step1 <- step(stepwise_model,
scope = list(lower = formula(forward_model),
upper = formula(back_model)),
direction = "both")
, stepwise_step1
sw_model <- lm(Crime ~ M + Ed + Po1 + M.F + U1 + U2 + Ineq + Prob, data
= crime)
summary(sw_model)
library(DAAG)
c <- cv.lm(crime,sw_model,m=5)
# Total Sum of Squared Diffs
sstot <- sum((crime$Crime - mean(crime$Crime))^2)
ss_model <- sum(stepwise_model$residuals^2)
ss_sw_model <- sum(sw_model$residuals^2)
ss_c <- attr(c, "ms")*nrow(crime)
#R^2 values
1 - ss_model/sstot # original model 0.803
1 - ss_sw_model/sstot # stepwise model 0.788
1 - ss_c/sstot # cross validated model 0.585
LASSO
The LASSO model was created using glmnet with an alpha value of 1 (indicating a LASSO
model, L1 is only considered in order to shrink the coefficients to zero), 10 folds, gaussian
and standardize set to true in order to scale the data. The lambda with the lowest mean-
squared error was 5.443807. It’s coefficients eliminated Po2, LF and Time. The log(lambda)
vs MSE is shown below the coefficients.
Crime ~ M + So + Ed + Po1 + M.F + Pop + NW + U1 + U2 + Wealth + Ineq + Prob
(Intercept) -5.614205e+03
M 7.884235e+01
So 3.709319e+01
Ed 1.445945e+02
Po1 1.003014e+02
M.F 1.846316e+01
Pop -2.609277e-01
NW 1.098298e+00
U1 -3.265759e+03
U2 1.186912e+02
Wealth 3.733655e-02
Ineq 5.720408e+01
Question 11.1
Using the crime data set uscrime.txt from Questions 8.2, 9.1, and 10.1, build a regression
model using:
1. Stepwise regression
2. Lasso
3. Elastic net
For Parts 2 and 3, remember to scale the data first – otherwise, the regression coefficients
will be on different scales and the constraint won’t have the desired effect. For Parts 2 and 3,
use the glmnet function in R.
TL;DR R^2 Values for the following models:
Plain, ole Linear Regression: 0.803
Stepwise: 0.788
Stepwise CV: 0.585
LASSO: 0.797
LASSO CV: 0.505
EN (0.25): 0.798
EN (0.25) CV: 0.544
Stepwise Regression:
The step function was used with a forward and backward model for the lower and
upper formulas. The step found with the lowest AIC is below, resulting in the formula:
Crime = -6426.10102 + 93.32155* M + 180.12011*Ed + 102.65316*Po1 + 22.33975*M.F
- 6086.63315* U1 + 187.34512*U2 + 61.33494*Ineq - 3796.03183*Prob
Stepwise regression found these variables to be insignificant to reducing AIC: Wealth, Pop,
Po2, So, LF, NW, or Time.
Step: AIC=503.93
Crime ~ M + Ed + Po1 + M.F + U1 + U2 + Ineq + Prob
Df Sum of Sq RSS AIC
<none> 1453068 503.93
+ Wealth 1 26493 1426575 505.07
- M.F 1 103159 1556227 505.16
+ Pop 1 16697 1436371 505.39
+ Po2 1 14148 1438919 505.47
+ So 1 9329 1443739 505.63
,+ LF 1 4374 1448694 505.79
+ NW 1 3799 1449269 505.81
+ Time 1 2293 1450775 505.86
- U1 1 127044 1580112 505.87
- Prob 1 247978 1701046 509.34
- U2 1 255443 1708511 509.55
- M 1 296790 1749858 510.67
- Ed 1 445788 1898855 514.51
- Ineq 1 738244 2191312 521.24
- Po1 1 1672038 3125105 537.93
> stepwise_step1
Call:
lm(formula = Crime ~ M + Ed + Po1 + M.F + U1 + U2 + Ineq +
Prob, data = crime)
Coefficients:
(Intercept) M Ed Po1 M.F U1
U2
-6426.10 93.32 180.12 102.65 22.34 -6086.63
187.35
Ineq Prob
61.33 -3796.03
The p-values were less than 0.05 for the intercept, M, Ed, Po1, U2, Ineq and Prob, indicating
statistical significance of these factors. MF and U1 were the only ones that did not.
Call:
lm(formula = Crime ~ M + Ed + Po1 + M.F + U1 + U2 + Ineq + Prob,
data = crime)
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -6426.10 1194.61 -5.379 4.04e-06 ***
M 93.32 33.50 2.786 0.00828 **
Ed 180.12 52.75 3.414 0.00153 **
Po1 102.65 15.52 6.613 8.26e-08 ***
M.F 22.34 13.60 1.642 0.10874
U1 -6086.63 3339.27 -1.823 0.07622 .
U2 187.35 72.48 2.585 0.01371 *
Ineq 61.33 13.96 4.394 8.63e-05 ***
Prob -3796.03 1490.65 -2.547 0.01505 *
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’0.1‘’1
These results were then compared to the original, full regression model and a 5 -fold cross
validated model’s R^2 values. The original model was 0.803. The stepwise model was 0.788.
,The cross-validated model was 0.585. The cross validated predicted values per fold are
shown below, along with the code. Full code and results-in-line are in the appendix.
Unset
rm(list = ls())
set.seed(5)
crime <- read.table("~/DocumentsLocal/GT/ISYE-
6501/hw7/uscrime.txt", stringsAsFactors = FALSE, header = TRUE)
# Backward elimination
back_model <- lm(Crime~., data = crime)
# Forward selection
forward_model <- lm(Crime~1, data = crime)
# Stepwise regression
stepwise_model <- lm(Crime~., data = crime)
stepwise_step1 <- step(stepwise_model,
scope = list(lower = formula(forward_model),
upper = formula(back_model)),
direction = "both")
, stepwise_step1
sw_model <- lm(Crime ~ M + Ed + Po1 + M.F + U1 + U2 + Ineq + Prob, data
= crime)
summary(sw_model)
library(DAAG)
c <- cv.lm(crime,sw_model,m=5)
# Total Sum of Squared Diffs
sstot <- sum((crime$Crime - mean(crime$Crime))^2)
ss_model <- sum(stepwise_model$residuals^2)
ss_sw_model <- sum(sw_model$residuals^2)
ss_c <- attr(c, "ms")*nrow(crime)
#R^2 values
1 - ss_model/sstot # original model 0.803
1 - ss_sw_model/sstot # stepwise model 0.788
1 - ss_c/sstot # cross validated model 0.585
LASSO
The LASSO model was created using glmnet with an alpha value of 1 (indicating a LASSO
model, L1 is only considered in order to shrink the coefficients to zero), 10 folds, gaussian
and standardize set to true in order to scale the data. The lambda with the lowest mean-
squared error was 5.443807. It’s coefficients eliminated Po2, LF and Time. The log(lambda)
vs MSE is shown below the coefficients.
Crime ~ M + So + Ed + Po1 + M.F + Pop + NW + U1 + U2 + Wealth + Ineq + Prob
(Intercept) -5.614205e+03
M 7.884235e+01
So 3.709319e+01
Ed 1.445945e+02
Po1 1.003014e+02
M.F 1.846316e+01
Pop -2.609277e-01
NW 1.098298e+00
U1 -3.265759e+03
U2 1.186912e+02
Wealth 3.733655e-02
Ineq 5.720408e+01
