Reliability: Conceptual Basis
Classical Test Theory: reliability reflects the extent to which the
differences in respondents’ test scores are a function of their true
psychological differences, as opposed to measurement error.
Continuum of reliability, not yes/no issue
Reliability is an unobserved feature
Classical test theory (CTT)
Observed scores: measurement of characteristics of a
person
True scores: real amounts of a characteristic(inter or intra
individual difference) average score that a participant would
obtain if he or she completed the scale an infinite number of
times.
o Scores that could arise if measurement/test was
perfectly precise
Measurement error: extent to which these “other”
characteristics contribute to differences in observed score
o Assumption: error occurs as if it is random, which
partially means that the error is as likely to inflate a
score as it is to decrease it.
o The influence of an error to someone’s score is
independent of the true scores.
o Average effect of error across respondents is zero
o Correlation of true scores and error is zero
Reliability: the extent to which differences in respondents’
observed scores are consistent with differences in their true
score as opposed to unknown administration characteristics.
XO = Xt + Xe
Variances of Observed scores, True & Error scores
, Variance of error: degree to which error affected different
ppl in different ways. (high variance, poor measurement)
Variance of Observed score: So2 = St2 + Se2 (1)
o Composite score: So2= St2 +Se2 + 2rteStSe
→ The correlation between true and error scores is zero and
that is why we end up with formula (1)
Four ways to think of Reliability
1. Reliability: True score variance/ Observed score
variance
2
St
Rxx΄ = 2
So
That number indicates the % of the differences that we see
among respondents’ observed scores can be attributed to
differences among their individual traits.
Range between 0 & 1, if variance is of true scores is 0 >
reliability is 0, if true score variance is equal to the
observed score variance > reliability is 1.
<.5 = low, form .7-.8 and larger = satisfying
2. Reliability: Lack of Error variance
Classical Test Theory: reliability reflects the extent to which the
differences in respondents’ test scores are a function of their true
psychological differences, as opposed to measurement error.
Continuum of reliability, not yes/no issue
Reliability is an unobserved feature
Classical test theory (CTT)
Observed scores: measurement of characteristics of a
person
True scores: real amounts of a characteristic(inter or intra
individual difference) average score that a participant would
obtain if he or she completed the scale an infinite number of
times.
o Scores that could arise if measurement/test was
perfectly precise
Measurement error: extent to which these “other”
characteristics contribute to differences in observed score
o Assumption: error occurs as if it is random, which
partially means that the error is as likely to inflate a
score as it is to decrease it.
o The influence of an error to someone’s score is
independent of the true scores.
o Average effect of error across respondents is zero
o Correlation of true scores and error is zero
Reliability: the extent to which differences in respondents’
observed scores are consistent with differences in their true
score as opposed to unknown administration characteristics.
XO = Xt + Xe
Variances of Observed scores, True & Error scores
, Variance of error: degree to which error affected different
ppl in different ways. (high variance, poor measurement)
Variance of Observed score: So2 = St2 + Se2 (1)
o Composite score: So2= St2 +Se2 + 2rteStSe
→ The correlation between true and error scores is zero and
that is why we end up with formula (1)
Four ways to think of Reliability
1. Reliability: True score variance/ Observed score
variance
2
St
Rxx΄ = 2
So
That number indicates the % of the differences that we see
among respondents’ observed scores can be attributed to
differences among their individual traits.
Range between 0 & 1, if variance is of true scores is 0 >
reliability is 0, if true score variance is equal to the
observed score variance > reliability is 1.
<.5 = low, form .7-.8 and larger = satisfying
2. Reliability: Lack of Error variance
