Furr, & Bacharach (2014) – Psychometrics An introduction
Chapter 11 – Test Bias
- Test biases can systematically obscure the differences (or lack thereof) among groups
of respondents can have important personal + societal implications (e.g., want to
know if gender differences exist in math ability if bias, then difference in gender
cannot be interpreted correctly).
- Group difference in test scores does not necessarily mean that test scores are biased.
- 2 types of bias (construct + predictive bias), which are independent of each other.
- Why worry about test score bias? Scores on psychological tests are often used to make
important decisions about people, e.g., about education in need of tests that
produces scores that allow to differentiate among people based on real psychological
differences, not on group membership. Rooted in our belief that we shouldn’t
discriminate.
Detecting construct bias: internal evaluation of a test
- Construct bias: bias in the meaning of a test (= measurement bias/internal bias).
Concerns the relationship of observed to true scores: if difference is systematically
different for different groups, then test might be biased.
- E.g., mechanical aptitude test suffering from construct bias related to biological sex:
test scores might reflect different psychological attributes in the 2 groups, for
example, males’ responses determined primarily by single construct mechanical
aptitude, but females’ responses determined by 2 constructs, mechanical aptitude +
stereotype threat test doesn’t measure same psychological attributes for 2 sexes,
so inappropriate to compare scores and say that men have higher mechanical
aptitude.
- Evaluated by examining responses to individual items on a test. Item is biased if:
o People belonging to different groups responded to different ways to the item;
o AND if it could be shown that differing responses were not related to group
differences associated with psychological attribute measured by test.
- If ≥ 1 items seem to be biased, then suspect total test score to be biased, because it’s
a composite score.
- About reflecting true scores; can never be known, so using estimates.
- Different procedures that focus on internal structure of the test (= the way parts of a
test are related to each other = pattern of correlations).
- Evaluate construct bias by examining internal structure of test separately for 2 groups
same internal structure in groups = test unlikely to suffer from construct bias.
- Methods for detecting construct bias:
a. Item discrimination index: reflects degree to which item is related to total test
score (i.e., that people who answer item correctly tend to do better on whole
test).
Strong discrimination = item is highly similar conceptually to most of
other items = reflecting associations among test items.
Originally for classical test theory, to measure extent to which
responses to items can be used to differentiate among people.
Proportion of people answering question correctly can be used to
compute an item discrimination index: having high level of construct =
high probability of answering correctly/high; low level = low probability
1
Chapter 11 – Test Bias
- Test biases can systematically obscure the differences (or lack thereof) among groups
of respondents can have important personal + societal implications (e.g., want to
know if gender differences exist in math ability if bias, then difference in gender
cannot be interpreted correctly).
- Group difference in test scores does not necessarily mean that test scores are biased.
- 2 types of bias (construct + predictive bias), which are independent of each other.
- Why worry about test score bias? Scores on psychological tests are often used to make
important decisions about people, e.g., about education in need of tests that
produces scores that allow to differentiate among people based on real psychological
differences, not on group membership. Rooted in our belief that we shouldn’t
discriminate.
Detecting construct bias: internal evaluation of a test
- Construct bias: bias in the meaning of a test (= measurement bias/internal bias).
Concerns the relationship of observed to true scores: if difference is systematically
different for different groups, then test might be biased.
- E.g., mechanical aptitude test suffering from construct bias related to biological sex:
test scores might reflect different psychological attributes in the 2 groups, for
example, males’ responses determined primarily by single construct mechanical
aptitude, but females’ responses determined by 2 constructs, mechanical aptitude +
stereotype threat test doesn’t measure same psychological attributes for 2 sexes,
so inappropriate to compare scores and say that men have higher mechanical
aptitude.
- Evaluated by examining responses to individual items on a test. Item is biased if:
o People belonging to different groups responded to different ways to the item;
o AND if it could be shown that differing responses were not related to group
differences associated with psychological attribute measured by test.
- If ≥ 1 items seem to be biased, then suspect total test score to be biased, because it’s
a composite score.
- About reflecting true scores; can never be known, so using estimates.
- Different procedures that focus on internal structure of the test (= the way parts of a
test are related to each other = pattern of correlations).
- Evaluate construct bias by examining internal structure of test separately for 2 groups
same internal structure in groups = test unlikely to suffer from construct bias.
- Methods for detecting construct bias:
a. Item discrimination index: reflects degree to which item is related to total test
score (i.e., that people who answer item correctly tend to do better on whole
test).
Strong discrimination = item is highly similar conceptually to most of
other items = reflecting associations among test items.
Originally for classical test theory, to measure extent to which
responses to items can be used to differentiate among people.
Proportion of people answering question correctly can be used to
compute an item discrimination index: having high level of construct =
high probability of answering correctly/high; low level = low probability
1
