STAT 4230/6230 — Regression Analysis
Personal Study Notes
Chapter 2 – Foundations of Regression Analysis
What Is Regression?
The central goal of regression is to figure out how one variable (called the response or
dependent variable, y) relates to one or more other variables (called predictors or
independent variables, x₁, x₂, …). Once that relationship is modeled, we can use new x-
values to predict future y-values.
Quick terminology:
• Response / dependent variable (y): what we want to predict — can be
quantitative or categorical
• Predictor / independent / explanatory variable (x): what we use to do the
predicting — also can be quantitative or categorical
Why Can't We Just Use the Mean?
If we know nothing about any predictor variable, the best single guess for y is its sample
mean ȳ. But that ignores all the useful information the x-variables carry, leaving large
prediction errors. Regression does better by adjusting predictions based on x.
The formal way to write this: yᵢ = E(y) + εᵢ, where E(y) is what the model predicts and ε
captures everything the model misses (unexplained random variation).
Types of Studies
Observational Studies
The x-values are measured as they naturally occur — we don't control or manipulate
them. These are by far the most common. The key limitation: we cannot draw cause-
and-effect conclusions from observational data, because hidden 'lurking' variables might
be driving the relationship we observe.
Experimental Studies
The researcher deliberately assigns or controls x-values (treatments). If participants are
randomly assigned to conditions, any differences in y can be traced back to the
treatment — so cause-and-effect conclusions are valid.
, Example to think about: A study finds that people who drink green tea daily live longer.
Is this observational or experimental? Could a lurking variable (healthier overall lifestyle)
explain this? What would an experiment need to look like to actually prove green tea
causes longer life?
Modeling is Iterative
We don't pick the 'right' model on the first try. The process typically goes: start with a
simple linear model → check if assumptions are satisfied → if not, try transformations,
interaction terms, or higher-order terms → repeat until the model fits well.
Chapter 3 – Simple Linear Regression
Ch. 3.2 – The Simple Linear Model
When we believe y has a straight-line relationship with a single predictor x, we write:
E(y) = β₀ + β₁x
The term E(y) represents what the model predicts on average. Because no model is
perfect, actual observed values scatter around E(y) due to a random error term ε. A
good model keeps ε small — meaning most of the variation in y is explained by x.
Interpreting the coefficients:
• Slope (β₁): the expected change in y for each one-unit increase in x. If β₁ = 0, the
line is horizontal and x tells us nothing about y.
• Intercept (β₀): the predicted value of y when x = 0. Only interpret this if x = 0 is
actually possible and makes sense in context. Extending predictions outside the
range of your data (extrapolation) is risky — the trend may not continue.
Ch. 3.3 – Least Squares (OLS)
To fit a line we need a rule for what 'best fit' means. Ordinary Least Squares (OLS)
chooses β₀ and β₁ to minimize the total squared distance from each data point to the
line — the Sum of Squared Errors (SSE).
The resulting formulas (derived by taking partial derivatives and setting them to zero):
• β̂₁ = SSxy / SSxx where SSxy = Σ(xᵢ - x̄)(yᵢ - ȳ) and SSxx = Σ(xᵢ - x̄)²
• β̂₀ = ȳ - β̂₁x̄
Why square the errors instead of using absolute values? Squaring is mathematically
convenient (differentiable everywhere) and penalizes large errors more heavily. The
Personal Study Notes
Chapter 2 – Foundations of Regression Analysis
What Is Regression?
The central goal of regression is to figure out how one variable (called the response or
dependent variable, y) relates to one or more other variables (called predictors or
independent variables, x₁, x₂, …). Once that relationship is modeled, we can use new x-
values to predict future y-values.
Quick terminology:
• Response / dependent variable (y): what we want to predict — can be
quantitative or categorical
• Predictor / independent / explanatory variable (x): what we use to do the
predicting — also can be quantitative or categorical
Why Can't We Just Use the Mean?
If we know nothing about any predictor variable, the best single guess for y is its sample
mean ȳ. But that ignores all the useful information the x-variables carry, leaving large
prediction errors. Regression does better by adjusting predictions based on x.
The formal way to write this: yᵢ = E(y) + εᵢ, where E(y) is what the model predicts and ε
captures everything the model misses (unexplained random variation).
Types of Studies
Observational Studies
The x-values are measured as they naturally occur — we don't control or manipulate
them. These are by far the most common. The key limitation: we cannot draw cause-
and-effect conclusions from observational data, because hidden 'lurking' variables might
be driving the relationship we observe.
Experimental Studies
The researcher deliberately assigns or controls x-values (treatments). If participants are
randomly assigned to conditions, any differences in y can be traced back to the
treatment — so cause-and-effect conclusions are valid.
, Example to think about: A study finds that people who drink green tea daily live longer.
Is this observational or experimental? Could a lurking variable (healthier overall lifestyle)
explain this? What would an experiment need to look like to actually prove green tea
causes longer life?
Modeling is Iterative
We don't pick the 'right' model on the first try. The process typically goes: start with a
simple linear model → check if assumptions are satisfied → if not, try transformations,
interaction terms, or higher-order terms → repeat until the model fits well.
Chapter 3 – Simple Linear Regression
Ch. 3.2 – The Simple Linear Model
When we believe y has a straight-line relationship with a single predictor x, we write:
E(y) = β₀ + β₁x
The term E(y) represents what the model predicts on average. Because no model is
perfect, actual observed values scatter around E(y) due to a random error term ε. A
good model keeps ε small — meaning most of the variation in y is explained by x.
Interpreting the coefficients:
• Slope (β₁): the expected change in y for each one-unit increase in x. If β₁ = 0, the
line is horizontal and x tells us nothing about y.
• Intercept (β₀): the predicted value of y when x = 0. Only interpret this if x = 0 is
actually possible and makes sense in context. Extending predictions outside the
range of your data (extrapolation) is risky — the trend may not continue.
Ch. 3.3 – Least Squares (OLS)
To fit a line we need a rule for what 'best fit' means. Ordinary Least Squares (OLS)
chooses β₀ and β₁ to minimize the total squared distance from each data point to the
line — the Sum of Squared Errors (SSE).
The resulting formulas (derived by taking partial derivatives and setting them to zero):
• β̂₁ = SSxy / SSxx where SSxy = Σ(xᵢ - x̄)(yᵢ - ȳ) and SSxx = Σ(xᵢ - x̄)²
• β̂₀ = ȳ - β̂₁x̄
Why square the errors instead of using absolute values? Squaring is mathematically
convenient (differentiable everywhere) and penalizes large errors more heavily. The
