Simple Linear Regression : Response Var (y): random. var (x). Points should be symmetrically distributed, tending to grps) , Independence, Normality. Linearity not needed (ANOVA
Predicting Var (x): Fixed, does not chg w response, but it is set cluster to middle of the plot, and have no clear pattern/trend in is w.r.t categorical var). Pooled variance estimator S2=
fixed before the response is measured. Regression is used for: 1) residuals; 2) Constant variance (Residuals against fitted values k ni
Prediction of y, 2) Modelling r/s between response and (y). Should see a horizontal band pattern around zero line. ∑∑ (Y ij
2
- Ý )
explanatory var, 3) Testing hypothesis of r/s. Megaphone effect (increasing/decreasing) shows violation to i=1 j=1
=SST ; Sampling dist of the pooled var
constant variance assumption. 3) Independence. Residual N-k N-k
analysis cannot be used to check independence assumption. is chi-sq with n-k df. SST = SSE + SSTR. RecallSST=
Instead, we can assess whether the residuals are uncorrelated. k ni k ni 2
(e.g clustering shows correlation). 4) Normality (Q-Q plot, ∑∑ ( Y ∑∑ ( Y - Y^´ i ) ; SSR=
2
ij - Y´ ) ; SSE = ij
histogram). Straight line shows normality. Use histogram to i=1 j=1 i=1 j=1
To estimate (b0,b1): Min SS diff bet observed and expected complement Q-Q plot to check normality. Residuals should have n
values of response var. Given estimates of (b0,b1), we define: 1) symmetric distribution, unimodal, no gaps. Remedy: Power
^ ^ transformation of y (Box-Cox) to variance stabilize response var
∑ ( ^y −´y )
i
2
; where y_ij is the observed target value, y-
i=1
Fitted values: ^yi = β0 + β 1 xi, 2) Residuals: ri = (if normality or constant var doesn’t hold). Lambda: 1 (don’t bar is the mean value and y-hat_i is the predicted value. >>>>>
ε^ i = yi - ^y , 3) Mean squared error (variance estimator transform); -1 (inverse y), 0 (log y), 0.5 (sqrt y) Outliers; 1) MSE = SSE/(n-k): within-grp variability; 2) MSSTR = SSTR/
n
∑ ri2 Leverage pt; far from mean of x’s. Influential pt; far from the (k-1): btw grp; 3) ANOVA: comparing bet to within variability;
σ ): MSE =
2
=
SSE mean of either x/y and if they influence the fit of the regression. 4) F-test: Mean sum of sq treatments (MSReg) divide by MSE.
. (chi-squared
i=1
n-2 ¿ y i − ^y i Compares var bet grps against var within grps. If F-test is large,
n-2 Check outliers (look at standardized residuals r= var bet grp is larger than
i
distribution with n-2 df. Lose 2 df because b0 & b1 is replaced √MSE Multiple Linear Regression:Y ¿ X β+ ε ,whr
with their estimators) Interpretation of reg coeff: b1 is the and compare to -2 to +2 band or even tighter -1 to +1). Cook’s
−1 T
^β = ( X T X )0 Xwhen
approx. Y n is very large. (‘Inflated’ statistical
estimated expected change in y assoc. with one unit change of x. distance: measures how muchththe estimated parameter values in
b0 is the estimated expected value of y when x=0. Estimator for the reg model changes with i observation is removed. Rule of MSE( σ^ )=ε^ ε^ / (n−sub-sampling,
significance).
2 Remedy:
T anddist
p−1) , chi-sq apply
w re(n-p
-g1)mdofd. el to
reg coeff: E(β _hat)= β (unbiased). Var(β _hat)= σ 2
/S . i.e Thumb: D > 4/n, D > 1, OR any “large” D each sample
1 1 1 xx i i i Interpretation of coeff: bj is the estimated expected change in y
β1_hat ~Normal(β1, σ 2/Sxx). Replacing σ 2 with MSE, sampling Y^ (i) −Y^ ¿T (Y^ ( i)−Y^ ) assoc. with one unit change of x, holding all other predictors in
dist of β1_hat becomes t-dist with n-2 df. CI for b1: Given ¿ the model fixed. (Conditional of all other predicting var,
should be investigated. ( ). Model compared to SLR tt is marginal); can differ in magnitude & sign.
sampling dist of b1, we can derive CI → ¿ Types of predicting var: 1) Controlling – to control for bias
^β t MSE Di=¿
. Testing significance: Reject H0:b1=0 vs. selection, used as default var to capture meaningful r/s; 2)
1 α
2
,n−2 √ Sxx performance: 1) Coeff of determination R2=1-SSE/SST, Explanatory – to explain variability in y; 3) Predictive – to
β^ 1 √SXX best predict variability in y regardless of explanatory power.
where SSE=∑n r2 ; SST =∑
n ( y −y)
2
. R2 shows
HA: b1 ≠ 0 if t-value ( )> critical pt ta/2,n-2 or if p- i i
σ^ i=1 i=1
the proportion of total variability in Y that can be explained by value 2xPr(t(n-
∝ = 2)>|t0|)<
, ; i.e the predictor (and reg model) the regression (that uses x). 2) Correlation coeff
ρ to see
is significant. If we want to test for positive (negative) r/s: H0:b1
how well the X’s are linearly related to y. ρ =R ANOVA:
2 2
<= (>=) 0, vs HA:b1 > (<) 0. Test p-value > (<) t-value, right (left)
th
dist of b1. Prediction: Variability Due to (n+1) observation within To compare the means across k population. Box-plots: Observe
variability (within each box) to the between variability of
(new measurement) & estimation(of b0, b1), and some due to
the response data (variability between means proxied by middle Testing overall regression (F-test): H0: β
random error. Confidence vs prediction intervals: 1) Prediction line in the boxplot). significant differences across the means if
1¿ .. βp =0 vs. HA: at
bands are wider due to higher uncertainty in predictions, 2) the between-variability is larger than the within-variability. least one , β ≠ 0 . Reject H0 if F-statistic is large (> Fα, p, n-p-1
p
Intervals increases towards the end due to higher uncertainty as Primary objectives:1) Analysis of variability in the data using
we move away from data average, 3) Intervals gets narrower as ANOVA table, 2) Equal means. H0: all means are equal. H :
o
sample size (n) increases (smaller variability). *PI(CI) is used to
provide an interval estimate for y^ for one (all) member of the μ 1 = μ 2 = … = μ k vs HA:at least two of the means for α significance level, p and n-p-1 degrees of freedom). Reject
are not equal. 3) Pairwise comparison by est.CI for all the pairs H0 if p-value is small: P-value = Prob(Fp, n-p-1 > F-statistic). If H0 is
pop. with a particular value of x*.Model assumptions
(Goodness of fit): 1) Linearity (Residuals (e) against predicting of means. H0:means in the pair are equal vs HA that are not rejected, then at least one of the coeff is different from zero at
equal. Assumptions: Constant variance (plot ri by treatment the α sig. level, hence overall reg is statistically significant.
Predicting Var (x): Fixed, does not chg w response, but it is set cluster to middle of the plot, and have no clear pattern/trend in is w.r.t categorical var). Pooled variance estimator S2=
fixed before the response is measured. Regression is used for: 1) residuals; 2) Constant variance (Residuals against fitted values k ni
Prediction of y, 2) Modelling r/s between response and (y). Should see a horizontal band pattern around zero line. ∑∑ (Y ij
2
- Ý )
explanatory var, 3) Testing hypothesis of r/s. Megaphone effect (increasing/decreasing) shows violation to i=1 j=1
=SST ; Sampling dist of the pooled var
constant variance assumption. 3) Independence. Residual N-k N-k
analysis cannot be used to check independence assumption. is chi-sq with n-k df. SST = SSE + SSTR. RecallSST=
Instead, we can assess whether the residuals are uncorrelated. k ni k ni 2
(e.g clustering shows correlation). 4) Normality (Q-Q plot, ∑∑ ( Y ∑∑ ( Y - Y^´ i ) ; SSR=
2
ij - Y´ ) ; SSE = ij
histogram). Straight line shows normality. Use histogram to i=1 j=1 i=1 j=1
To estimate (b0,b1): Min SS diff bet observed and expected complement Q-Q plot to check normality. Residuals should have n
values of response var. Given estimates of (b0,b1), we define: 1) symmetric distribution, unimodal, no gaps. Remedy: Power
^ ^ transformation of y (Box-Cox) to variance stabilize response var
∑ ( ^y −´y )
i
2
; where y_ij is the observed target value, y-
i=1
Fitted values: ^yi = β0 + β 1 xi, 2) Residuals: ri = (if normality or constant var doesn’t hold). Lambda: 1 (don’t bar is the mean value and y-hat_i is the predicted value. >>>>>
ε^ i = yi - ^y , 3) Mean squared error (variance estimator transform); -1 (inverse y), 0 (log y), 0.5 (sqrt y) Outliers; 1) MSE = SSE/(n-k): within-grp variability; 2) MSSTR = SSTR/
n
∑ ri2 Leverage pt; far from mean of x’s. Influential pt; far from the (k-1): btw grp; 3) ANOVA: comparing bet to within variability;
σ ): MSE =
2
=
SSE mean of either x/y and if they influence the fit of the regression. 4) F-test: Mean sum of sq treatments (MSReg) divide by MSE.
. (chi-squared
i=1
n-2 ¿ y i − ^y i Compares var bet grps against var within grps. If F-test is large,
n-2 Check outliers (look at standardized residuals r= var bet grp is larger than
i
distribution with n-2 df. Lose 2 df because b0 & b1 is replaced √MSE Multiple Linear Regression:Y ¿ X β+ ε ,whr
with their estimators) Interpretation of reg coeff: b1 is the and compare to -2 to +2 band or even tighter -1 to +1). Cook’s
−1 T
^β = ( X T X )0 Xwhen
approx. Y n is very large. (‘Inflated’ statistical
estimated expected change in y assoc. with one unit change of x. distance: measures how muchththe estimated parameter values in
b0 is the estimated expected value of y when x=0. Estimator for the reg model changes with i observation is removed. Rule of MSE( σ^ )=ε^ ε^ / (n−sub-sampling,
significance).
2 Remedy:
T anddist
p−1) , chi-sq apply
w re(n-p
-g1)mdofd. el to
reg coeff: E(β _hat)= β (unbiased). Var(β _hat)= σ 2
/S . i.e Thumb: D > 4/n, D > 1, OR any “large” D each sample
1 1 1 xx i i i Interpretation of coeff: bj is the estimated expected change in y
β1_hat ~Normal(β1, σ 2/Sxx). Replacing σ 2 with MSE, sampling Y^ (i) −Y^ ¿T (Y^ ( i)−Y^ ) assoc. with one unit change of x, holding all other predictors in
dist of β1_hat becomes t-dist with n-2 df. CI for b1: Given ¿ the model fixed. (Conditional of all other predicting var,
should be investigated. ( ). Model compared to SLR tt is marginal); can differ in magnitude & sign.
sampling dist of b1, we can derive CI → ¿ Types of predicting var: 1) Controlling – to control for bias
^β t MSE Di=¿
. Testing significance: Reject H0:b1=0 vs. selection, used as default var to capture meaningful r/s; 2)
1 α
2
,n−2 √ Sxx performance: 1) Coeff of determination R2=1-SSE/SST, Explanatory – to explain variability in y; 3) Predictive – to
β^ 1 √SXX best predict variability in y regardless of explanatory power.
where SSE=∑n r2 ; SST =∑
n ( y −y)
2
. R2 shows
HA: b1 ≠ 0 if t-value ( )> critical pt ta/2,n-2 or if p- i i
σ^ i=1 i=1
the proportion of total variability in Y that can be explained by value 2xPr(t(n-
∝ = 2)>|t0|)<
, ; i.e the predictor (and reg model) the regression (that uses x). 2) Correlation coeff
ρ to see
is significant. If we want to test for positive (negative) r/s: H0:b1
how well the X’s are linearly related to y. ρ =R ANOVA:
2 2
<= (>=) 0, vs HA:b1 > (<) 0. Test p-value > (<) t-value, right (left)
th
dist of b1. Prediction: Variability Due to (n+1) observation within To compare the means across k population. Box-plots: Observe
variability (within each box) to the between variability of
(new measurement) & estimation(of b0, b1), and some due to
the response data (variability between means proxied by middle Testing overall regression (F-test): H0: β
random error. Confidence vs prediction intervals: 1) Prediction line in the boxplot). significant differences across the means if
1¿ .. βp =0 vs. HA: at
bands are wider due to higher uncertainty in predictions, 2) the between-variability is larger than the within-variability. least one , β ≠ 0 . Reject H0 if F-statistic is large (> Fα, p, n-p-1
p
Intervals increases towards the end due to higher uncertainty as Primary objectives:1) Analysis of variability in the data using
we move away from data average, 3) Intervals gets narrower as ANOVA table, 2) Equal means. H0: all means are equal. H :
o
sample size (n) increases (smaller variability). *PI(CI) is used to
provide an interval estimate for y^ for one (all) member of the μ 1 = μ 2 = … = μ k vs HA:at least two of the means for α significance level, p and n-p-1 degrees of freedom). Reject
are not equal. 3) Pairwise comparison by est.CI for all the pairs H0 if p-value is small: P-value = Prob(Fp, n-p-1 > F-statistic). If H0 is
pop. with a particular value of x*.Model assumptions
(Goodness of fit): 1) Linearity (Residuals (e) against predicting of means. H0:means in the pair are equal vs HA that are not rejected, then at least one of the coeff is different from zero at
equal. Assumptions: Constant variance (plot ri by treatment the α sig. level, hence overall reg is statistically significant.
