Regularized regression
The following practical presents an exercise on multiple linear regression
models, regularization and validation.
1 Student data set
The data were obtained in a survey of students in portuguese language
courses in secondary school. It contains a lot of interesting social, gender and
study information about students.
It contains the following attributes:
1. school - student’s school (binary: ’GP’ - Gabriel Pereira or ’MS’ - Mous-
inho da Silveira)
2. sex - student’s sex (binary: "F" - female or "M" - male)
3. age - student’s age (numeric: from 15 to 22)
4. address - student’s home address type (binary: "U" - urban or "R" - rural)
5. famsize - family size (binary: "LE3" - less or equal to 3 or "GT3" -
greater than 3)
6. Pstatus - parent’s cohabitation status (binary: "T" - living together or "A"
- apart)
7. Medu - mother’s education (numeric: 0 - none, 1 - primary education
(4th grade), 2 - 5th to 9th grade, 3 - secondary education or 4 - higher
education)
8. Fedu - father’s education (numeric: 0 - none, 1 - primary education (4th
grade), 2 - 5th to 9th grade, 3 - secondary education or 4 - higher educa-
tion)
9. Mjob - mother’s job (nominal: "teacher", "health" care related, civil "ser-
vices" (e.g. administrative or police), "athome" or "other")
10. Fjob - father’s job (nominal: "teacher", "health" care related, civil "ser-
vices" (e.g. administrative or police), "athome" or "other")
11. reason - reason to choose this school (nominal: close to "home", school
"reputation", "course" preference or "other")
12. guardian - student’s guardian (nominal: "mother", "father" or "other")
13. traveltime - home to school travel time (numeric: 1 - <15 min., 2 - 15 to
30 min., 3 - 30 min. to 1 hour, or 4 - >1 hour)
14. studytime - weekly study time (numeric: 1 - <2 hours, 2 - 2 to 5 hours, 3
- 5 to 10 hours, or 4 - >10 hours)
15. failures - number of past class failures (numeric: n if 1<=n<3, else 4)
16. schoolsup - extra educational support (binary: yes or no)
1
, 17. famsup - family educational support (binary: yes or no)
18. paid - extra paid classes within the course subject (Math or Portuguese)
(binary: yes or no)
19. activities - extra-curricular activities (binary: yes or no)
20. nursery - attended nursery school (binary: yes or no)
21. higher - wants to take higher education (binary: yes or no)
22. internet - Internet access at home (binary: yes or no)
23. romantic - with a romantic relationship (binary: yes or no)
24. famrel - quality of family relationships (numeric: from 1 - very bad to 5 -
excellent)
25. freetime - free time after school (numeric: from 1 - very low to 5 - very
high)
26. goout - going out with friends (numeric: from 1 - very low to 5 - very
high)
27. Dalc - workday alcohol consumption (numeric: from 1 - very low to 5 -
very high)
28. Walc - weekend alcohol consumption (numeric: from 1 - very low to 5 -
very high)
29. health - current health status (numeric: from 1 - very bad to 5 - very
good)
30. absences - number of school absences (numeric: from 0 to 93)
We can read in this data with the following command.
students<-read.csv("student-por.csv")
There are three scores included (G1, G2 for the midterms and G3 for the final
total). We will only concern ourselves with predicting the final total in this
exercise. Therefore let’s start with removing the 2 penultimate columns.
data <- students[,-
c(31,32)] summary(data)
1.1 Dummy variables
If you check the summary statistics of the data set, you will notice that there
are many categorical variables, such as gender, school, etc. We are going to
recode these into numerical features for use within our regression model. We
will do this by creating a new variable into each possibility (for example
gender-male and gender-female) and assigning a 0 or a 1. These new variables
are called ’dummy variables’. However we will only retain all but one of these
expanded variables to avoid redundancy in the dataset. For example, if we
know gender- female, then we know gender-male as well. Luckily there is an
2
The following practical presents an exercise on multiple linear regression
models, regularization and validation.
1 Student data set
The data were obtained in a survey of students in portuguese language
courses in secondary school. It contains a lot of interesting social, gender and
study information about students.
It contains the following attributes:
1. school - student’s school (binary: ’GP’ - Gabriel Pereira or ’MS’ - Mous-
inho da Silveira)
2. sex - student’s sex (binary: "F" - female or "M" - male)
3. age - student’s age (numeric: from 15 to 22)
4. address - student’s home address type (binary: "U" - urban or "R" - rural)
5. famsize - family size (binary: "LE3" - less or equal to 3 or "GT3" -
greater than 3)
6. Pstatus - parent’s cohabitation status (binary: "T" - living together or "A"
- apart)
7. Medu - mother’s education (numeric: 0 - none, 1 - primary education
(4th grade), 2 - 5th to 9th grade, 3 - secondary education or 4 - higher
education)
8. Fedu - father’s education (numeric: 0 - none, 1 - primary education (4th
grade), 2 - 5th to 9th grade, 3 - secondary education or 4 - higher educa-
tion)
9. Mjob - mother’s job (nominal: "teacher", "health" care related, civil "ser-
vices" (e.g. administrative or police), "athome" or "other")
10. Fjob - father’s job (nominal: "teacher", "health" care related, civil "ser-
vices" (e.g. administrative or police), "athome" or "other")
11. reason - reason to choose this school (nominal: close to "home", school
"reputation", "course" preference or "other")
12. guardian - student’s guardian (nominal: "mother", "father" or "other")
13. traveltime - home to school travel time (numeric: 1 - <15 min., 2 - 15 to
30 min., 3 - 30 min. to 1 hour, or 4 - >1 hour)
14. studytime - weekly study time (numeric: 1 - <2 hours, 2 - 2 to 5 hours, 3
- 5 to 10 hours, or 4 - >10 hours)
15. failures - number of past class failures (numeric: n if 1<=n<3, else 4)
16. schoolsup - extra educational support (binary: yes or no)
1
, 17. famsup - family educational support (binary: yes or no)
18. paid - extra paid classes within the course subject (Math or Portuguese)
(binary: yes or no)
19. activities - extra-curricular activities (binary: yes or no)
20. nursery - attended nursery school (binary: yes or no)
21. higher - wants to take higher education (binary: yes or no)
22. internet - Internet access at home (binary: yes or no)
23. romantic - with a romantic relationship (binary: yes or no)
24. famrel - quality of family relationships (numeric: from 1 - very bad to 5 -
excellent)
25. freetime - free time after school (numeric: from 1 - very low to 5 - very
high)
26. goout - going out with friends (numeric: from 1 - very low to 5 - very
high)
27. Dalc - workday alcohol consumption (numeric: from 1 - very low to 5 -
very high)
28. Walc - weekend alcohol consumption (numeric: from 1 - very low to 5 -
very high)
29. health - current health status (numeric: from 1 - very bad to 5 - very
good)
30. absences - number of school absences (numeric: from 0 to 93)
We can read in this data with the following command.
students<-read.csv("student-por.csv")
There are three scores included (G1, G2 for the midterms and G3 for the final
total). We will only concern ourselves with predicting the final total in this
exercise. Therefore let’s start with removing the 2 penultimate columns.
data <- students[,-
c(31,32)] summary(data)
1.1 Dummy variables
If you check the summary statistics of the data set, you will notice that there
are many categorical variables, such as gender, school, etc. We are going to
recode these into numerical features for use within our regression model. We
will do this by creating a new variable into each possibility (for example
gender-male and gender-female) and assigning a 0 or a 1. These new variables
are called ’dummy variables’. However we will only retain all but one of these
expanded variables to avoid redundancy in the dataset. For example, if we
know gender- female, then we know gender-male as well. Luckily there is an
2
