CHAPTER 4 RELIABILITY
DEFINING RELIABILITY
For an instrument to have sound psychometric characteristics, it should have test reliability.
Reliability = the consistency with which it measures whatever it measures
To obtain R measures we have to apply 3 aspects consistently:
1. What the entity is that we are measuring eg weight/height
2. What the nature of the measure is eg ruler, scale
3. Application of the rules on how to measure the object eg with/without clothes or shoes
The goal of estimating reliability is:
Determine how much of the variability in test scores is due to errors in measurement and how much
is due to variability in true scores.
True score = the replicable feature of the concept being measured. It is the part of the observed
score that would recur across different measurement occasions in the absence of error.
Errors of measurement = composed of both random error and systematic error. Represents the
discrepancies between scores obtained on tests and the corresponding true scores.
We never know a person’s true score so we use observed data to compute the reliability of tests.
Reliability coefficient = numerical expression for reliability
TYPES OF RELIABILITY
1)Test-retest reliability 2) Alternate-form reliability 3) Split half reliability
4) Inter-item consistency(Kuder-Richardson & Cronbach Alpha) 5) Inter-scorer reliability
1)Test-retest reliability:
= Repeating the test to the same group on a different occasion
The reliability coefficient (rc) is correlation between 1st & 2nd test also known as coefficient of
stability
The interval between retests should not exceed 6 months so that the changes are random and not
cumulative or progressive
Disadvantages:
- Test takers circumstances change eg illness, emotions, fatigue…
1
, - The physical environment alters eg venue, weather, noise….
- Transfer effects:
o Practice could equate to improvement of score
o Short interval between re test may result in recall
Generally not appropriate for computing reliability.
2) Alternate-form reliability
= Administering an alternate form of the same test to the same group on a different occasion
The RC is correlation between 1st & 2nd test also known as coefficient of equivalence (if
administered close together). If administered several days apart it’s known as coefficient of stability
& equivalence.
Difference between the scores illustrates the type of error variance under consideration.
The 2 measures need to be truly equivalent eg same # of items, scoring procedure…..
Disadvantages:
• Difficult to create several alternate forms of a test
• Difficult if not impossible to guarantee that two alternate forms of a test are parallel measures
• Not available for many tests due to practicality of producing equivalent forms
3) Split-half reliability
= Administering a test to a group of individuals and then splitting the test in half and correlating scores
on one half of the test with scores on the other half.
The RC is correlation between two split halves also known as coefficient of internal consistency.
This halves reliability. Estimate is then stepped up to the full test length using the Spearman–Brown
prediction formula.
Most common way to split a test is to take odd and even items not the 1st & the 2nd halves as tests do
get harder.
4) Inter-item consistency(Kuder-Richardson & Cronbach Alpha)
= based on the consistency of responses to all items in the measure
To calculate the reliability of item scores of 1 or 0 (dichotomous) you use Kudar-Richardson 20
(KR20)
2
DEFINING RELIABILITY
For an instrument to have sound psychometric characteristics, it should have test reliability.
Reliability = the consistency with which it measures whatever it measures
To obtain R measures we have to apply 3 aspects consistently:
1. What the entity is that we are measuring eg weight/height
2. What the nature of the measure is eg ruler, scale
3. Application of the rules on how to measure the object eg with/without clothes or shoes
The goal of estimating reliability is:
Determine how much of the variability in test scores is due to errors in measurement and how much
is due to variability in true scores.
True score = the replicable feature of the concept being measured. It is the part of the observed
score that would recur across different measurement occasions in the absence of error.
Errors of measurement = composed of both random error and systematic error. Represents the
discrepancies between scores obtained on tests and the corresponding true scores.
We never know a person’s true score so we use observed data to compute the reliability of tests.
Reliability coefficient = numerical expression for reliability
TYPES OF RELIABILITY
1)Test-retest reliability 2) Alternate-form reliability 3) Split half reliability
4) Inter-item consistency(Kuder-Richardson & Cronbach Alpha) 5) Inter-scorer reliability
1)Test-retest reliability:
= Repeating the test to the same group on a different occasion
The reliability coefficient (rc) is correlation between 1st & 2nd test also known as coefficient of
stability
The interval between retests should not exceed 6 months so that the changes are random and not
cumulative or progressive
Disadvantages:
- Test takers circumstances change eg illness, emotions, fatigue…
1
, - The physical environment alters eg venue, weather, noise….
- Transfer effects:
o Practice could equate to improvement of score
o Short interval between re test may result in recall
Generally not appropriate for computing reliability.
2) Alternate-form reliability
= Administering an alternate form of the same test to the same group on a different occasion
The RC is correlation between 1st & 2nd test also known as coefficient of equivalence (if
administered close together). If administered several days apart it’s known as coefficient of stability
& equivalence.
Difference between the scores illustrates the type of error variance under consideration.
The 2 measures need to be truly equivalent eg same # of items, scoring procedure…..
Disadvantages:
• Difficult to create several alternate forms of a test
• Difficult if not impossible to guarantee that two alternate forms of a test are parallel measures
• Not available for many tests due to practicality of producing equivalent forms
3) Split-half reliability
= Administering a test to a group of individuals and then splitting the test in half and correlating scores
on one half of the test with scores on the other half.
The RC is correlation between two split halves also known as coefficient of internal consistency.
This halves reliability. Estimate is then stepped up to the full test length using the Spearman–Brown
prediction formula.
Most common way to split a test is to take odd and even items not the 1st & the 2nd halves as tests do
get harder.
4) Inter-item consistency(Kuder-Richardson & Cronbach Alpha)
= based on the consistency of responses to all items in the measure
To calculate the reliability of item scores of 1 or 0 (dichotomous) you use Kudar-Richardson 20
(KR20)
2
