Chapter 6: The Beast of Bias
Points illustrated by the population model
- The form of the model is similar in the samples to the population
- Parameter estimates vary across samples (there is sampling variation): on average the
parameters should reflect the true population value
- The spread of scores around the population model is consistent
- The population model contains (independent) error: for each value of the predictor, the
errors are modelled with a normal distribution = a probability distribution of a random
variable that is known to have certain properties, such as a mean of 0 and a constant
variance: it is symmetrical (skew of 0) and has a kurtosis of 3 or excess kurtosis of 0
o For any predicted value the error associated with that value is, in average, 0:
predicted values are thus assumed to have no error
Residual = the difference between the value a model predicts and the value
observed in the data on which the model is based (basically, an error but in a
sample instead of population)
Residuals can be observed and thus used to infer things about errors
in prediction for the population model which can’t be observed
Deviations = when the mean is used as a model residuals are called
deviations
To assess the error in a linear model one must use a sum of squared
errors: residual sum of squares, RSS = a measure of the variability
that cannot be explain by the model fitted to the data: the total
squared deviance between the observations, and the value of those
observations predicted by whatever model is fitted to the data
o Large RSS indicates the model is unrepresentative of the
data
o Small RSS indicates the model is representative of the data
o Ordinary least squares, OLS = uses mathematical techniques
for finding minima to find the b-values that minimuze the
RSS for the model (it is the maximum likelihood estimate)
o Estimate of variance of errors = RSS / df
When model errors are normally
distributed (in small samples), the
sampling distribution of the estimated
parameter is also normal, which allows
for estimating the standard error of the
estimated parameter and confidence
intervals and test statistics
o The sampling distribution is also normal for large samples (n
of at least 30): as samples get larger, sample means tend to
be the same as the population mean and are normally
distributed with variance equal to the population variance
divided by the sample size (central limit theorem applies to
any parameter, not just the mean)
Gauss-Markov theorem = states the OLS estimate is the best of the relevant estimators when certain
conditions are met:
- Model errors are, on average, 0: the expected value of model errors is 0
- Homoscedasticity (homogeneity of variance) = an assumption that the variance of one
variable is stable at all levels of another variable
- Independent errors = for any two observations in regression the residuals should be
uncorrelated
,Parameter estimates that are at odds with the true values
- Unbiased estimator = a statistical estimator that is expected to equal the true value of the
parameter it estimates, meaning its average value over many samples equals the parameter
being estimated
- Consistent estimator = an estimator that, as the sample size increases, converges towards
the true value of the parameter being estimated
- Efficient estimator = an estimator that has the smallest possible variance among all unbiased
estimators of a parameter
- Coverage = the proportion of times that the confidence intervals, if computed from many
different samples, would contain the true parameter value
Sources of bias
- Outlier = an observation very different from most others, which biases statistics, standard
errors, and confidence intervals: affects the sum of squared error more dramatically than the
parameter itself (as squared errors uses squaring, thus magnifying the deviations and
outliers)
o Mixed (contaminated) normal distribution = a normal-looking distribution that is
contaminated by a small proportion of scores from a different distribution: these
distributions are not normal and have too many scores in the tails, the effect of these
heavy tails is to inflate the estimate of the population variance, which makes the
significance tests lack power
- Violations of assumptions (in order of importance, most to least):
o Assumption of linearity and additivity
Scores on the outcome variable are linearly related to any predictors, and
that if one has several predictors, then their combined effect is best
described by adding their effects together
Tested by plotting a Residuals vs Predicted plot: if the scatterplot follow a
straight line and residuals scatter randomly around 0 then the assumption is
met
o Assumption that the expected value of errors is 0
o Assumption of spherical errors
Homoscedasticity: when violated there is heteroscedasticity (heterogeneity
of variance) = the variance of one variable varies across levels of another
variable
If it’s violated use weighted least squares = a method of regression
in which the parameters of the model are estimated using the
method of least squares but observations are weighted by some
other variable: often they are weighted by the inverse of their
variance to combat heteroscedasticity
o It will produce inaccurate standard errors, confidence
intervals, and significance tests when homoscedasticity is
violated
o If will produce unbiased estimates of the model parameters
when homoscedasticity is violated, but other methods will
do so with less error
Homogeneity can be tested using Levene’s test = this tests the
hypothesis that the variances in different groups are equal: a
significant result indicates that the variances are significantly
different, therefore the assumption of homoscedasticity has been
violated
, o When sample sizes are large, small differences in group
variances can produce a significant Levene’s test (this test is
not recommended)
Homogeneity can be tested using Brown-Forsythe test = a version of
the F-statistic designed to be accurate when the assumption of
homoscedasticity has been violated
Homogeneity can be tested using Hartley’s FMax (variance ratio) =
ratio of the variances between the group with the biggest variance
and the group with the smallest variance: the ratio is compared to
critical values in a table as a test of homoscedasticity
o With samples of 10 per group, an F Max less than 10 is more or
less always going to be non-significant
o With samples of 15-20 per group, an FMax needs to be less
than about 5
o With samples of 30-60, an FMax should be below about 2 or 3
Independence of errors
Always violated in data collected from the same entities over time
When violated use statistical models that factor in dependence
between errors
o Assumption of normality = a common misunderstanding is that this assumption
relates to the observed data but when using ordinary least squares the assumption
relates to the errors of the model, not the observed data
When errors cannot be assumed to have a normal distribution, model
parameter estimates are unbiased
For the standard errors of model parameters to be unbiased one needs
normally distributed errors or a large sample size otherwise confidence
intervals and significance tests or the model parameters estimates will be
biased (because they both use the standard error in their calculation)
It is the least important assumption:
Parameter estimation does not require normality: method of least
squares will always give an estimate of the model parameters that
minimizes error and is unbiased
Normality doesn’t matter in large samples
Normality doesn’t matter when residuals are normally distributed
Test normality by looking at Q-Q (quantile-quantile) plot = a graph plotting
the quantiles of a variable against the quantiles of a particular distribution: if
values fall on the diagonal of the plot then the variable shares the same
distribution as the one specified, and deviations from the diagonal show
deviations from the distribution of interest
Test normality by looking at P-P (probability-probability) plot = a graph
plotting the cumulative probability of a variable against the cumulative
probability of a particular distribution: if values fall on the diagonal of the
plot then the variable shares the same distribution as the one specified, and
deviations from the diagonal show deviations from the distribution of
interest
Test normality by using significance tests (only in small samples!):
Kolmogorov-Smirnov test = a test of whether a distribution of scores
is significantly different from a normal distribution: a significant
value indicates a deviation from normality, but this test is
notoriously affected by large samples in which small deviations from
normality yield signifcant results
Points illustrated by the population model
- The form of the model is similar in the samples to the population
- Parameter estimates vary across samples (there is sampling variation): on average the
parameters should reflect the true population value
- The spread of scores around the population model is consistent
- The population model contains (independent) error: for each value of the predictor, the
errors are modelled with a normal distribution = a probability distribution of a random
variable that is known to have certain properties, such as a mean of 0 and a constant
variance: it is symmetrical (skew of 0) and has a kurtosis of 3 or excess kurtosis of 0
o For any predicted value the error associated with that value is, in average, 0:
predicted values are thus assumed to have no error
Residual = the difference between the value a model predicts and the value
observed in the data on which the model is based (basically, an error but in a
sample instead of population)
Residuals can be observed and thus used to infer things about errors
in prediction for the population model which can’t be observed
Deviations = when the mean is used as a model residuals are called
deviations
To assess the error in a linear model one must use a sum of squared
errors: residual sum of squares, RSS = a measure of the variability
that cannot be explain by the model fitted to the data: the total
squared deviance between the observations, and the value of those
observations predicted by whatever model is fitted to the data
o Large RSS indicates the model is unrepresentative of the
data
o Small RSS indicates the model is representative of the data
o Ordinary least squares, OLS = uses mathematical techniques
for finding minima to find the b-values that minimuze the
RSS for the model (it is the maximum likelihood estimate)
o Estimate of variance of errors = RSS / df
When model errors are normally
distributed (in small samples), the
sampling distribution of the estimated
parameter is also normal, which allows
for estimating the standard error of the
estimated parameter and confidence
intervals and test statistics
o The sampling distribution is also normal for large samples (n
of at least 30): as samples get larger, sample means tend to
be the same as the population mean and are normally
distributed with variance equal to the population variance
divided by the sample size (central limit theorem applies to
any parameter, not just the mean)
Gauss-Markov theorem = states the OLS estimate is the best of the relevant estimators when certain
conditions are met:
- Model errors are, on average, 0: the expected value of model errors is 0
- Homoscedasticity (homogeneity of variance) = an assumption that the variance of one
variable is stable at all levels of another variable
- Independent errors = for any two observations in regression the residuals should be
uncorrelated
,Parameter estimates that are at odds with the true values
- Unbiased estimator = a statistical estimator that is expected to equal the true value of the
parameter it estimates, meaning its average value over many samples equals the parameter
being estimated
- Consistent estimator = an estimator that, as the sample size increases, converges towards
the true value of the parameter being estimated
- Efficient estimator = an estimator that has the smallest possible variance among all unbiased
estimators of a parameter
- Coverage = the proportion of times that the confidence intervals, if computed from many
different samples, would contain the true parameter value
Sources of bias
- Outlier = an observation very different from most others, which biases statistics, standard
errors, and confidence intervals: affects the sum of squared error more dramatically than the
parameter itself (as squared errors uses squaring, thus magnifying the deviations and
outliers)
o Mixed (contaminated) normal distribution = a normal-looking distribution that is
contaminated by a small proportion of scores from a different distribution: these
distributions are not normal and have too many scores in the tails, the effect of these
heavy tails is to inflate the estimate of the population variance, which makes the
significance tests lack power
- Violations of assumptions (in order of importance, most to least):
o Assumption of linearity and additivity
Scores on the outcome variable are linearly related to any predictors, and
that if one has several predictors, then their combined effect is best
described by adding their effects together
Tested by plotting a Residuals vs Predicted plot: if the scatterplot follow a
straight line and residuals scatter randomly around 0 then the assumption is
met
o Assumption that the expected value of errors is 0
o Assumption of spherical errors
Homoscedasticity: when violated there is heteroscedasticity (heterogeneity
of variance) = the variance of one variable varies across levels of another
variable
If it’s violated use weighted least squares = a method of regression
in which the parameters of the model are estimated using the
method of least squares but observations are weighted by some
other variable: often they are weighted by the inverse of their
variance to combat heteroscedasticity
o It will produce inaccurate standard errors, confidence
intervals, and significance tests when homoscedasticity is
violated
o If will produce unbiased estimates of the model parameters
when homoscedasticity is violated, but other methods will
do so with less error
Homogeneity can be tested using Levene’s test = this tests the
hypothesis that the variances in different groups are equal: a
significant result indicates that the variances are significantly
different, therefore the assumption of homoscedasticity has been
violated
, o When sample sizes are large, small differences in group
variances can produce a significant Levene’s test (this test is
not recommended)
Homogeneity can be tested using Brown-Forsythe test = a version of
the F-statistic designed to be accurate when the assumption of
homoscedasticity has been violated
Homogeneity can be tested using Hartley’s FMax (variance ratio) =
ratio of the variances between the group with the biggest variance
and the group with the smallest variance: the ratio is compared to
critical values in a table as a test of homoscedasticity
o With samples of 10 per group, an F Max less than 10 is more or
less always going to be non-significant
o With samples of 15-20 per group, an FMax needs to be less
than about 5
o With samples of 30-60, an FMax should be below about 2 or 3
Independence of errors
Always violated in data collected from the same entities over time
When violated use statistical models that factor in dependence
between errors
o Assumption of normality = a common misunderstanding is that this assumption
relates to the observed data but when using ordinary least squares the assumption
relates to the errors of the model, not the observed data
When errors cannot be assumed to have a normal distribution, model
parameter estimates are unbiased
For the standard errors of model parameters to be unbiased one needs
normally distributed errors or a large sample size otherwise confidence
intervals and significance tests or the model parameters estimates will be
biased (because they both use the standard error in their calculation)
It is the least important assumption:
Parameter estimation does not require normality: method of least
squares will always give an estimate of the model parameters that
minimizes error and is unbiased
Normality doesn’t matter in large samples
Normality doesn’t matter when residuals are normally distributed
Test normality by looking at Q-Q (quantile-quantile) plot = a graph plotting
the quantiles of a variable against the quantiles of a particular distribution: if
values fall on the diagonal of the plot then the variable shares the same
distribution as the one specified, and deviations from the diagonal show
deviations from the distribution of interest
Test normality by looking at P-P (probability-probability) plot = a graph
plotting the cumulative probability of a variable against the cumulative
probability of a particular distribution: if values fall on the diagonal of the
plot then the variable shares the same distribution as the one specified, and
deviations from the diagonal show deviations from the distribution of
interest
Test normality by using significance tests (only in small samples!):
Kolmogorov-Smirnov test = a test of whether a distribution of scores
is significantly different from a normal distribution: a significant
value indicates a deviation from normality, but this test is
notoriously affected by large samples in which small deviations from
normality yield signifcant results
