Summary of the material for the final exam (2022) for Statistics II: Applied Quantitative Analysis.
INCLUDES notes from (Total: 42 pages):
● Abby Long’s guide (2016) “10 Things to Know About Reading a Regression Table” for
EGAP’s website.
● Tom Louwerse’s guide “Reporting regression analysis” for Universiteit Leiden Statistics II:
Applied Quantitative Analysis.
● Tom Louwerse’s guide “Reporting logistic regression analysis” for Universiteit Leiden
Statistics II: Applied Quantitative Analysis.
● Andy Field’s book (5th edition, 2018) “Discovering Statistics Using IBM SPSS Statistics”,
chapters 6, 9 (sections 9.1, 9.2 and 9.4-9.11), 11 (sections 11.3-11.3.5, 11.4-11.4.3 and 11.5)
and 20 (20.1-20.6 and 20.8).
1
Statistics II: Applied Quantitative Analysis Notes on Readings
Table of Contents
“10 Things to Know About Reading a Regression Table” 2
“Reporting regression analysis” 6
“Reporting logistic regression analysis” 8
“Discovering Statistics Using IBM SPSS Statistics” 9
6. The Beast of Bias 9
9. The Linear Model (Regression) 16
11. Moderation, Mediation and Multicategory Predictors 31
20. Categorical Outcomes: Logistic Regression 36
, 2
“10 Things to Know About Reading a Regression Table”
1. What is regression?
Regression: A method for calculating the line of best fit, using the “independent variables” to predict
the outcome or “dependent variable”.
➔ The DV = the output/response.
➔ The IV = inputs/predictors or variables tested to see if they predict the outcome.
Regression table steps:
1. Figure out what the DV is.
2. Identify the most important IVs.
3. Base interpretation on these.
Relationships:
● Positive relationship = When the IV increases, the DV tends to increase.
● Negative relationship = When the IV increases, the DV tends to decrease, and vice versa.
2. What is a regression equation?
The regression formula, where:
● 𝑌 = Dependent variable.
● α = Intercept (Alpha parameter); the predicted value of Y when 𝑋 = 0.
● β = Slope (Beta parameter); the predicted change in Y for each one-unit increase in X.
● ε = “Error term”; the remaining variation in Y that CANNOT be explained by a linear
relationship with X.
𝑌 = α + β𝑋 + ε
Multiple Regression: A regression with one DV and more than one IV (can include quadratic or other
nonlinear transformations). It predicts the value of the DV using several IVs.
3. What are the main purposes of regression?
Purposes of regression are to:
1. Give a descriptive summary of how the outcome varies with the explanatory variables.
➔ Ordinary Least Squares (OLS): Regression showing the best-fitting linear
relationship (minimising the sum of squares of the residuals). If there is a sufficiently
large sample that was randomly drawn from a much larger population, OLS estimates
the best-fitting line in the population and allows the use of estimated coefficients and
“robust” standard errors to construct confidence intervals for the coefficients of the
population line.
◆ Sum of Squares of the Residuals: The differences between the actual
outcomes and the values predicted from the explanatory variables.
➔ The OLS summary may miss the data’s important features (e.g. outliers or nonlinear
relationships).
2. Predict the outcome, given a set of values for the explanatory variables.
, 3
➔ OLS regression gives the best linear predictor in the sample, and if the sample is
drawn randomly from a larger population, OLS is a consistent estimator of the
population’s best linear predictor. HOWEVER:
◆ The best linear predictor from a particular set of regressors may NOT be the
best predictor that can be constructed from the available data.
◆ A prediction that works well in our sample or similar populations may NOT
work well in other populations.
3. Estimate the parameters of a model describing a process that generates the outcome.
4. Study causal relationships by limiting the use of covariates to reduce bias (in an
observational study with strong assumptions) or variance (in a randomised experiment with
weaker assumptions).
➔ Covariates: Other explanatory variables.
4. What are the standard errors, t-statistics, p-values, and degrees of freedom?
Standard Error (SE): An estimate of the standard deviation of an estimated coefficient (i.e. a
measure of the precision with which the regression coefficient is estimated). Important for the
construction of confidence intervals (CIs) and significance tests
➔ Small SE = more precise estimate of the coefficient.
➔ Rule of thumb = when the sample is reasonably large, the margin of error for a 95% CI is
approximately twice the SE.
➔ The key assumptions that “conventional/classical” SEs make and robust SEs relax are that
the:
1. Expected value of Y, given X, is a linear function of X.
2. Variance of Y does NOT depend on X (conditional homoscedasticity).
3. Robust SEs assume (unless “clustered”) either that the observations are statistically
independent or that the treatment was randomly assigned to the units of observation.
t-Statistic: The ratio of the estimated coefficient to its standard error. Used as a tool for
constructing CIs and significance tests.
Estimates are “statistically significant” at the 1%, 5%, or 10% level (i.e. if the p-value from a
two-sided test of the null hypothesis where the true coefficient is 0) is below 0.01, 0.05, or 0.10.
➔ p-Values: The probability that the t-statistic’s absolute value ≥ observed value if the true
coefficient were 0. If the p-value ≥ some conventional threshold (e.g. 0.05 or 0.10), the
estimate is “not statistically significant” (at the 5% or 10% level).
➔ Intercept: The predicted value of the outcome when the values of the explanatory variables
are all 0.
➔ Estimates that are NOT statistically significant are NOT considered evidence that the true
coefficient is nonzero.
◆ It is easy to misinterpret p-values and significance tests.
F-Test: A test of the null hypothesis that the true values of the regression coefficients, excluding the
intercept, are all 0 (i.e. the null hypothesis is that none of the explanatory variables actually help
predict the outcome).
Degrees of Freedom (df): The numbers in parentheses are associated with the F-statistic formula’s
numerator and denominator.
INCLUDES notes from (Total: 42 pages):
● Abby Long’s guide (2016) “10 Things to Know About Reading a Regression Table” for
EGAP’s website.
● Tom Louwerse’s guide “Reporting regression analysis” for Universiteit Leiden Statistics II:
Applied Quantitative Analysis.
● Tom Louwerse’s guide “Reporting logistic regression analysis” for Universiteit Leiden
Statistics II: Applied Quantitative Analysis.
● Andy Field’s book (5th edition, 2018) “Discovering Statistics Using IBM SPSS Statistics”,
chapters 6, 9 (sections 9.1, 9.2 and 9.4-9.11), 11 (sections 11.3-11.3.5, 11.4-11.4.3 and 11.5)
and 20 (20.1-20.6 and 20.8).
1
Statistics II: Applied Quantitative Analysis Notes on Readings
Table of Contents
“10 Things to Know About Reading a Regression Table” 2
“Reporting regression analysis” 6
“Reporting logistic regression analysis” 8
“Discovering Statistics Using IBM SPSS Statistics” 9
6. The Beast of Bias 9
9. The Linear Model (Regression) 16
11. Moderation, Mediation and Multicategory Predictors 31
20. Categorical Outcomes: Logistic Regression 36
, 2
“10 Things to Know About Reading a Regression Table”
1. What is regression?
Regression: A method for calculating the line of best fit, using the “independent variables” to predict
the outcome or “dependent variable”.
➔ The DV = the output/response.
➔ The IV = inputs/predictors or variables tested to see if they predict the outcome.
Regression table steps:
1. Figure out what the DV is.
2. Identify the most important IVs.
3. Base interpretation on these.
Relationships:
● Positive relationship = When the IV increases, the DV tends to increase.
● Negative relationship = When the IV increases, the DV tends to decrease, and vice versa.
2. What is a regression equation?
The regression formula, where:
● 𝑌 = Dependent variable.
● α = Intercept (Alpha parameter); the predicted value of Y when 𝑋 = 0.
● β = Slope (Beta parameter); the predicted change in Y for each one-unit increase in X.
● ε = “Error term”; the remaining variation in Y that CANNOT be explained by a linear
relationship with X.
𝑌 = α + β𝑋 + ε
Multiple Regression: A regression with one DV and more than one IV (can include quadratic or other
nonlinear transformations). It predicts the value of the DV using several IVs.
3. What are the main purposes of regression?
Purposes of regression are to:
1. Give a descriptive summary of how the outcome varies with the explanatory variables.
➔ Ordinary Least Squares (OLS): Regression showing the best-fitting linear
relationship (minimising the sum of squares of the residuals). If there is a sufficiently
large sample that was randomly drawn from a much larger population, OLS estimates
the best-fitting line in the population and allows the use of estimated coefficients and
“robust” standard errors to construct confidence intervals for the coefficients of the
population line.
◆ Sum of Squares of the Residuals: The differences between the actual
outcomes and the values predicted from the explanatory variables.
➔ The OLS summary may miss the data’s important features (e.g. outliers or nonlinear
relationships).
2. Predict the outcome, given a set of values for the explanatory variables.
, 3
➔ OLS regression gives the best linear predictor in the sample, and if the sample is
drawn randomly from a larger population, OLS is a consistent estimator of the
population’s best linear predictor. HOWEVER:
◆ The best linear predictor from a particular set of regressors may NOT be the
best predictor that can be constructed from the available data.
◆ A prediction that works well in our sample or similar populations may NOT
work well in other populations.
3. Estimate the parameters of a model describing a process that generates the outcome.
4. Study causal relationships by limiting the use of covariates to reduce bias (in an
observational study with strong assumptions) or variance (in a randomised experiment with
weaker assumptions).
➔ Covariates: Other explanatory variables.
4. What are the standard errors, t-statistics, p-values, and degrees of freedom?
Standard Error (SE): An estimate of the standard deviation of an estimated coefficient (i.e. a
measure of the precision with which the regression coefficient is estimated). Important for the
construction of confidence intervals (CIs) and significance tests
➔ Small SE = more precise estimate of the coefficient.
➔ Rule of thumb = when the sample is reasonably large, the margin of error for a 95% CI is
approximately twice the SE.
➔ The key assumptions that “conventional/classical” SEs make and robust SEs relax are that
the:
1. Expected value of Y, given X, is a linear function of X.
2. Variance of Y does NOT depend on X (conditional homoscedasticity).
3. Robust SEs assume (unless “clustered”) either that the observations are statistically
independent or that the treatment was randomly assigned to the units of observation.
t-Statistic: The ratio of the estimated coefficient to its standard error. Used as a tool for
constructing CIs and significance tests.
Estimates are “statistically significant” at the 1%, 5%, or 10% level (i.e. if the p-value from a
two-sided test of the null hypothesis where the true coefficient is 0) is below 0.01, 0.05, or 0.10.
➔ p-Values: The probability that the t-statistic’s absolute value ≥ observed value if the true
coefficient were 0. If the p-value ≥ some conventional threshold (e.g. 0.05 or 0.10), the
estimate is “not statistically significant” (at the 5% or 10% level).
➔ Intercept: The predicted value of the outcome when the values of the explanatory variables
are all 0.
➔ Estimates that are NOT statistically significant are NOT considered evidence that the true
coefficient is nonzero.
◆ It is easy to misinterpret p-values and significance tests.
F-Test: A test of the null hypothesis that the true values of the regression coefficients, excluding the
intercept, are all 0 (i.e. the null hypothesis is that none of the explanatory variables actually help
predict the outcome).
Degrees of Freedom (df): The numbers in parentheses are associated with the F-statistic formula’s
numerator and denominator.
