CHAPTER 11. TEST BIAS 1
Test Bias
Test bias: systematic score difference between groups on a test that are not due
to true differences in the latent construct but to some property of the test.
Construct bias (internal or
measurement bias): the relationship
between observed scores and true
scores differs systematically for
different groups.
Predictive bias: A test’s use has
different implications for different
groups. The relationship between
scores on a test and a criterion differs
systematically for different groups.
The two biases are independent from each other, construct bias might exist
without predictive bias and vice versa. When both are present they are still
independent.
Detecting construct bias: internal evaluation of a
test
Construct bias: the meaning of the scores on a test differs for different groups of
people
Examining the responses to individual items of a test
Biased when both of the following points are true:
People that belong to different groups respond differently on the items.
These differences in the responses are not related to group differences
associated with the psychological attribute we want to measure.
Methods
1. Item discrimination index
2. Factor analysis: different factor structure for different groups
3. Differential item Functioning analysis
4. Rank order: based on item difficulty and see if the order is the same for
both groups.
, CHAPTER 11. TEST BIAS 2
Item Discrimination index
If they show a big difference in the indexes of the two groups it means that we have a
construct bias
If we find
the same
index for
both but the
two groups
have
differences
in their
answers,
one of
them is
superior,
there is no
bias
Differential item functioning
We look at the relationship between the item score and the underlying
construct level, if that differs for different groups. (Based on IRT pg.431)
ICC curve
The steepness of the line shows that there is a good discrimination ability
between scores of 0 and 3, whereas from -3 to 0 the ability of discrimination
is quite low.
This item is easier than the purple item, that is because it has greater
steepness
Uniform bias: Location bias because same shape but different
location
This item is different in shape and location from the purple item, people with
low true score have a higher probability to give correct answer and people
with high true scores have lower probability to score correctly.
Non-uniform bias: shape, or shape and location are different.
Test Bias
Test bias: systematic score difference between groups on a test that are not due
to true differences in the latent construct but to some property of the test.
Construct bias (internal or
measurement bias): the relationship
between observed scores and true
scores differs systematically for
different groups.
Predictive bias: A test’s use has
different implications for different
groups. The relationship between
scores on a test and a criterion differs
systematically for different groups.
The two biases are independent from each other, construct bias might exist
without predictive bias and vice versa. When both are present they are still
independent.
Detecting construct bias: internal evaluation of a
test
Construct bias: the meaning of the scores on a test differs for different groups of
people
Examining the responses to individual items of a test
Biased when both of the following points are true:
People that belong to different groups respond differently on the items.
These differences in the responses are not related to group differences
associated with the psychological attribute we want to measure.
Methods
1. Item discrimination index
2. Factor analysis: different factor structure for different groups
3. Differential item Functioning analysis
4. Rank order: based on item difficulty and see if the order is the same for
both groups.
, CHAPTER 11. TEST BIAS 2
Item Discrimination index
If they show a big difference in the indexes of the two groups it means that we have a
construct bias
If we find
the same
index for
both but the
two groups
have
differences
in their
answers,
one of
them is
superior,
there is no
bias
Differential item functioning
We look at the relationship between the item score and the underlying
construct level, if that differs for different groups. (Based on IRT pg.431)
ICC curve
The steepness of the line shows that there is a good discrimination ability
between scores of 0 and 3, whereas from -3 to 0 the ability of discrimination
is quite low.
This item is easier than the purple item, that is because it has greater
steepness
Uniform bias: Location bias because same shape but different
location
This item is different in shape and location from the purple item, people with
low true score have a higher probability to give correct answer and people
with high true scores have lower probability to score correctly.
Non-uniform bias: shape, or shape and location are different.
