Articles Test construction
Content
Article 1:
Hubley, A. M., & Zumbo, B. D. (2011). Validity and the consequences of test interpretation
and use. Social Indicators Research, 103, 219–230.
Article 2:
Jacobusse, G. W., Buuren S. van, Verkerk, P. H. (2006). An interval scale for development of
children aged 0-2 years. Statistics in Medicine, 25 (13), 2272- 2283.
Article 3:
Koster, M., Timmerman, M. E., Nakken, H., Pijl, S. J., & van Houten, E. J. (2009). Evaluating
social participation of pupils with special needs in regular primary schools:
Examination of a teacher questionnaire. European Journal of Psychological
Assessment, 25 (4), 213-222.
, Validity and the Consequences of Test Interpretation and Use (Hubley & Zumbo, 2011)
Abstract
The vast majority of measures have a purpose of personal and social change. If test developers and
users want measures to have personal and social consequences and impact, then it is critical to
consider the consequences and side effects of measurement in the validation process itself.
The purposes of this paper were to:
- Review Messick’s unified view of validity and clarify his consequential basis of test
interpretation (i.e., value implications) and use (i.e., social consequences).
- Discuss the kinds of questions evoked by value implications and social consequences and
their role in construct validity and score meaning.
- Present a reframing of Messick’s model and a new model of unified validity and validation.
- Bring the concept of multilevel measures under the same validation umbrella as individual
differences measures.
- Offer thoughts and directions for more explicit consideration of value implications, intended
social consequences, and unintended side effects of legitimate test interpretation and use.
Introduction
According to Messick, validity is an integrated evaluative judgment of the degree to which evidence
and rationales support the appropriateness of interpretations and actions based on test scores.
While many test developers and test users strive to examine Messick’s evidential basis for test
interpretation and use, they tend to fall short when it comes to thinking and writing about the
consequential basis of test interpretation and use. Some conclude that test developers and users
reject consequences as evidence in validity research, this article argues instead that most do not
know about or do not understand Messick’s characterization of consequences.
1 Unified Validity Theory
Validity is about:
- Inferences, interpretations, or decisions that are based on a test score and not the test itself.
- Whether the inference is useful given the sample and context in which they are working.
Under the unified view, validity is all about the construct and hence the meaning of scores. Key
elements of construct validation:
- Refer to types of validity evidence rather than distinct types of validity. Validity cannot rely
solely on any one of these complementary forms of evidence in isolation from the others.
o Six aspects of construct validity evidence: content, substantive processes, score
structure, generalizability, external relationships, and consequences of testing.
- Construct underrepresentation = the measure fails to include dimensions of the construct.
Construct-irrelevant variance = variance due to other distinct constructs are also present.
o They are always present to some extent; the goal is to minimize their presence.
- Validation is an ongoing process. The unified model provides a regulative ideal to strive for.
o But, since evidence is always incomplete, validation is a matter of making the most
reasonable case. This process is similar to the idea of repairing a ship while at sea.
Unified validity theory recognizes that values and cultural or societal norms change. The end result is
that we must continually validate the inferences we make.
Messick developed a model, the progressive matrix of validity, that emphasizes what one needs to
consider when validating inferences from measures. He organized his matrix in terms of function
(interpretation vs. use) and the basis for justifying validity (evidential basis vs. consequential basis).
Content
Article 1:
Hubley, A. M., & Zumbo, B. D. (2011). Validity and the consequences of test interpretation
and use. Social Indicators Research, 103, 219–230.
Article 2:
Jacobusse, G. W., Buuren S. van, Verkerk, P. H. (2006). An interval scale for development of
children aged 0-2 years. Statistics in Medicine, 25 (13), 2272- 2283.
Article 3:
Koster, M., Timmerman, M. E., Nakken, H., Pijl, S. J., & van Houten, E. J. (2009). Evaluating
social participation of pupils with special needs in regular primary schools:
Examination of a teacher questionnaire. European Journal of Psychological
Assessment, 25 (4), 213-222.
, Validity and the Consequences of Test Interpretation and Use (Hubley & Zumbo, 2011)
Abstract
The vast majority of measures have a purpose of personal and social change. If test developers and
users want measures to have personal and social consequences and impact, then it is critical to
consider the consequences and side effects of measurement in the validation process itself.
The purposes of this paper were to:
- Review Messick’s unified view of validity and clarify his consequential basis of test
interpretation (i.e., value implications) and use (i.e., social consequences).
- Discuss the kinds of questions evoked by value implications and social consequences and
their role in construct validity and score meaning.
- Present a reframing of Messick’s model and a new model of unified validity and validation.
- Bring the concept of multilevel measures under the same validation umbrella as individual
differences measures.
- Offer thoughts and directions for more explicit consideration of value implications, intended
social consequences, and unintended side effects of legitimate test interpretation and use.
Introduction
According to Messick, validity is an integrated evaluative judgment of the degree to which evidence
and rationales support the appropriateness of interpretations and actions based on test scores.
While many test developers and test users strive to examine Messick’s evidential basis for test
interpretation and use, they tend to fall short when it comes to thinking and writing about the
consequential basis of test interpretation and use. Some conclude that test developers and users
reject consequences as evidence in validity research, this article argues instead that most do not
know about or do not understand Messick’s characterization of consequences.
1 Unified Validity Theory
Validity is about:
- Inferences, interpretations, or decisions that are based on a test score and not the test itself.
- Whether the inference is useful given the sample and context in which they are working.
Under the unified view, validity is all about the construct and hence the meaning of scores. Key
elements of construct validation:
- Refer to types of validity evidence rather than distinct types of validity. Validity cannot rely
solely on any one of these complementary forms of evidence in isolation from the others.
o Six aspects of construct validity evidence: content, substantive processes, score
structure, generalizability, external relationships, and consequences of testing.
- Construct underrepresentation = the measure fails to include dimensions of the construct.
Construct-irrelevant variance = variance due to other distinct constructs are also present.
o They are always present to some extent; the goal is to minimize their presence.
- Validation is an ongoing process. The unified model provides a regulative ideal to strive for.
o But, since evidence is always incomplete, validation is a matter of making the most
reasonable case. This process is similar to the idea of repairing a ship while at sea.
Unified validity theory recognizes that values and cultural or societal norms change. The end result is
that we must continually validate the inferences we make.
Messick developed a model, the progressive matrix of validity, that emphasizes what one needs to
consider when validating inferences from measures. He organized his matrix in terms of function
(interpretation vs. use) and the basis for justifying validity (evidential basis vs. consequential basis).
