MATHEMATIC ST362: ST362/ST562 Lab 2 Notes: Simple and Multiple Linear Regression | 2026 Update

ST362/ST562 Lab 2 Notes: Simple and Multiple Linear Regression



1. Simple Linear Regression

Suppose we observe data


(𝑥1 , 𝑦1 ), (𝑥2 , 𝑦2 ), … , (𝑥𝑛 , 𝑦𝑛 ),

where 𝑥𝑖 is the observed value of an explanatory variable and 𝑦𝑖 is the observed value of a response
variable.
The simple linear regression model is


𝑌𝑖 = 𝛽0 + 𝛽1 𝑥𝑖 + 𝜀𝑖 , 𝑖 = 1, … , 𝑛,

where:

• 𝑌𝑖 is the random response variable;
• 𝑥𝑖 is treated as fixed or observed;
• 𝛽0 is the intercept parameter;
• 𝛽1 is the slope parameter;
• 𝜀𝑖 is the random error term.

The usual normal regression assumptions are


𝜀1 , … , 𝜀𝑛 are independent and 𝜀𝑖 ∼ 𝑁 (0, 𝜎2 ).

Equivalently,


𝑌𝑖 ∼ 𝑁 (𝛽0 + 𝛽1 𝑥𝑖 , 𝜎2 ),

and the responses 𝑌1 , … , 𝑌𝑛 are independent. Under this model,


𝐸(𝑌𝑖 ) = 𝛽0 + 𝛽1 𝑥𝑖 , Var(𝑌𝑖 ) = 𝜎2 .




1

, 1.1 Least Squares Estimators

The fitted regression line is


𝑦𝑖̂ = 𝛽0̂ + 𝛽1̂ 𝑥𝑖 .

The residual for observation 𝑖 is


𝑒𝑖 = 𝑦𝑖 − 𝑦𝑖̂ .

The least squares estimates are chosen to minimize the residual sum of squares

𝑛 𝑛
𝑆𝑆𝐸 = ∑(𝑦𝑖 − 𝑦𝑖̂ )2 = ∑(𝑦𝑖 − 𝛽0̂ − 𝛽1̂ 𝑥𝑖 )2 .
𝑖=1 𝑖=1

Define

1 𝑛 1 𝑛
𝑥̄ = ∑𝑥 , 𝑦̄ = ∑𝑦 ,
𝑛 𝑖=1 𝑖 𝑛 𝑖=1 𝑖

𝑛 𝑛
𝑆𝑥𝑥 = ∑(𝑥𝑖 − 𝑥)̄ 2 , 𝑆𝑥𝑦 = ∑(𝑥𝑖 − 𝑥)(𝑦
̄ 𝑖 − 𝑦).
̄
𝑖=1 𝑖=1

Then the least squares estimators are

𝑆𝑥𝑦
𝛽1̂ = , 𝛽0̂ = 𝑦 ̄ − 𝛽1̂ 𝑥.̄
𝑆𝑥𝑥

Thus the fitted regression equation is


𝑦 ̂ = 𝛽0̂ + 𝛽1̂ 𝑥.

The slope 𝛽1̂ is interpreted as the estimated change in the mean response for a one-unit increase in the
explanatory variable. The intercept 𝛽0̂ is the estimated mean response when 𝑥 = 0, provided that 𝑥 = 0
is meaningful in context.


1.2 Properties of the Least Squares Line

For a simple linear regression model with an intercept, the residuals satisfy

𝑛
∑ 𝑒𝑖 = 0
𝑖=1

and

𝑛
∑ 𝑥𝑖 𝑒𝑖 = 0.
𝑖=1


2