Modelling Procedures and Assumptions for Regression Models
Learning Objectives:
● Describe various model selection procedures for explanatory and exploratory research
● Name common pitfalls of various selection procedures
● Draw conclusions about the assumptions of a (multiple) regression model
● Identify influential observations
(1) Model Selection
We first discuss criteria for selecting a regression model by deciding which of a possibly large
collection of variables to include in the model
→ exploratory approach: ie finding a good set of explanatory variables
→ confirmatory (explanatory) approach: based on theory and hypotheses, to test a theoretical model
Types of model selection procedures:
1. Hypothesis-driven (explanatory) research
2. Exploratory research
3 Rules for Model Selection:
1. Include the relevant variables to make the model useful for theoretical purposes, so you can
address hypotheses posed by the study, with sensible control (and mediating) variables
2. Include enough variables to obtain good predictive power (ie include all effects that
contribute to R2)
3. Keep the model simple, avoiding unnecessary (higher-order) effects
○ Parsimonious principle: entities should not be multiplied beyond necessity
1.1 Explanatory Research
2 Options for Model Selection for Explanatory Research:
1. Model building
○ Start with effect for control variables
○ Add focal (ie main) predictor
○ Add hypothesised interaction-effect
2. Model trimming
○ Start with hypothesised (complete) model
○ If included, test if interaction-effect contributes to R2
Conclusively,
● If the model has interaction-effect, report the effect of the focal predictor at relevant levels of
the moderator (eg M - SD, M, and M + SD)
● If the model doesn't have interaction-effect, test and interpret the effect (b) of your focal
predictor
, 1.2 Exploratory Research
Model Selection for Exploratory Research
1. Identify relevant processes
2. Search for accurate predictions in practice
3. After explanatory research: what other factors play a role in this phenomena?
→ automated model selection is useful: 3 options of model selection
1. Backward elimination
○ Start with model with all predictor variables
○ Repeatedly remove variable that contributes least to R2
○ Stop if all included predictors have a significant (partial) effect on y (given
pre-determined a-level)
2. Forward selection
○ Start with a model without predictors
○ Repeatedly add variable that contribute most to R2
○ Stop when none of the remaining variables contributes significantly to the
predictive power of the model (given pre-determined a-level)
3. Stepwise regression
→ combination of backward and forward
○ Start with a model without predictors
○ Repeatedly add variables that contribute most to R2
○ In-between, remove variables that lost their (partial) explanatory power
(given pre-determined a-level)
○ Stop if none of the remaining variables contribute significantly to the
predictive power of the model (given pre-determined a-level)
Pitfalls of Automated Model Selection:
● These procedures do not always yield meaningful models (eg interaction-effect without main
effect)
● R2 in the sample ≠ R2 in the population
2 s 2 y −s2 s 2 y −MSE MSR
○ Alternative measure for R : R
2
adj = 2
= 2
=
s y s y s2 y
→ Corrects for the number of predictors in the model, and can decrease when we
add
Predictors
● Chance capitalisation: drawing a conclusion from data wholly or partly biassed in a particular
direction by chance
○ Use cross-validation to test predictive power outside the sample
Learning Objectives:
● Describe various model selection procedures for explanatory and exploratory research
● Name common pitfalls of various selection procedures
● Draw conclusions about the assumptions of a (multiple) regression model
● Identify influential observations
(1) Model Selection
We first discuss criteria for selecting a regression model by deciding which of a possibly large
collection of variables to include in the model
→ exploratory approach: ie finding a good set of explanatory variables
→ confirmatory (explanatory) approach: based on theory and hypotheses, to test a theoretical model
Types of model selection procedures:
1. Hypothesis-driven (explanatory) research
2. Exploratory research
3 Rules for Model Selection:
1. Include the relevant variables to make the model useful for theoretical purposes, so you can
address hypotheses posed by the study, with sensible control (and mediating) variables
2. Include enough variables to obtain good predictive power (ie include all effects that
contribute to R2)
3. Keep the model simple, avoiding unnecessary (higher-order) effects
○ Parsimonious principle: entities should not be multiplied beyond necessity
1.1 Explanatory Research
2 Options for Model Selection for Explanatory Research:
1. Model building
○ Start with effect for control variables
○ Add focal (ie main) predictor
○ Add hypothesised interaction-effect
2. Model trimming
○ Start with hypothesised (complete) model
○ If included, test if interaction-effect contributes to R2
Conclusively,
● If the model has interaction-effect, report the effect of the focal predictor at relevant levels of
the moderator (eg M - SD, M, and M + SD)
● If the model doesn't have interaction-effect, test and interpret the effect (b) of your focal
predictor
, 1.2 Exploratory Research
Model Selection for Exploratory Research
1. Identify relevant processes
2. Search for accurate predictions in practice
3. After explanatory research: what other factors play a role in this phenomena?
→ automated model selection is useful: 3 options of model selection
1. Backward elimination
○ Start with model with all predictor variables
○ Repeatedly remove variable that contributes least to R2
○ Stop if all included predictors have a significant (partial) effect on y (given
pre-determined a-level)
2. Forward selection
○ Start with a model without predictors
○ Repeatedly add variable that contribute most to R2
○ Stop when none of the remaining variables contributes significantly to the
predictive power of the model (given pre-determined a-level)
3. Stepwise regression
→ combination of backward and forward
○ Start with a model without predictors
○ Repeatedly add variables that contribute most to R2
○ In-between, remove variables that lost their (partial) explanatory power
(given pre-determined a-level)
○ Stop if none of the remaining variables contribute significantly to the
predictive power of the model (given pre-determined a-level)
Pitfalls of Automated Model Selection:
● These procedures do not always yield meaningful models (eg interaction-effect without main
effect)
● R2 in the sample ≠ R2 in the population
2 s 2 y −s2 s 2 y −MSE MSR
○ Alternative measure for R : R
2
adj = 2
= 2
=
s y s y s2 y
→ Corrects for the number of predictors in the model, and can decrease when we
add
Predictors
● Chance capitalisation: drawing a conclusion from data wholly or partly biassed in a particular
direction by chance
○ Use cross-validation to test predictive power outside the sample
