Task 3 Who is right?
Learning goals
1. What is validity?
a. Different forms of validity
b. How can you measure these forms?
2. How to test instruments on different forms of validity?
3. What can we do with the discrepancies that occur? (e.g. how you perceive
yourself, how others perceive you)
a. How can we explain these differences? (no straightforward answer)
Bron: Evers, A., Sijtsma, K., Lucassen, W., & Meijer, R. R. (2010)
Dutch Rating System for Test Quality
→ 7 criteria
1. Theoretical basis
2. Quality of the testing materials
3. Comprehensiveness of the manual
4. Norms
5. Reliability
6. Construct validity
7. Criterion validity
Validity
● Validity is the extent to which a test fulfils its purpose
● Differents forms of evidence on validity of tests should not be considered to represent
distinct types of validity
○ It’s a unitary concept
○ Important to collect evidence of validity that supports intended interpretation
● Standardized rating procedure is necessary to structure the concept of validity and
thereby the rating process
● Types of validity evidence that are construct-related are required for almost all tests,
whatever the purpose of the test use
● Types of validity evidence that are criterion-related will not be required for tests that
are not intended for prediction
Construct validity
● All research with respect to internal structure is considered relevant for construct validity
● Research concerning external structure is considered relevant for both construct and
criterion validity
● Evidence should support the claim: the test measures the intended trait or ability
○ So: it measures what it should measure
■ “What does the test measure?”
■ “Does the test measure the intended concept or does it partly or mainly
measure something else?”
, ● 6 types of research in support of construct validity
1. Research on the dimensionality of the item scores
Evidence based on factor analysis
● E.g. making measure for love
○ Items for love and liking
○ Fill out for loved one and friend
○ Factor analysis → you can see which items group together
■ In that way you can make categories
2. The psychometric quality of the items
3. Invariance of the factor structure and possible bias
→ supply evidence on internal structure (content)
4. Convergent and discriminant validity
5. Differences between relevant groups
6. Other research (e.g. research on criterion validity that is also relevant for
construct validity)
→ supply evidence on external structure
● Structure of items same as for reliability
○ First: provision of results is ascertained by means of a key item
○ Second: sufficiency of evidence supporting construct validity is assessed
○ Third: quality of research design is assessed
Criterion validity
● Research concerning external structure is considered relevant for both construct and
criterion validity
● Research on criterion-related evidence should demonstrate that a test score is a good
predictor of non-test behavior or outcome criteria
● Prediction can focus on
○ Past (retrospective validity)
○ Same moment in time (concurrent validity)
○ Future (predictive validity)
● Used in all tests → only: if test explicitly is not for prediction, than there is no need to
● Structure of items same as for construct validity
○ Recommendations contain a section on the use of ROC (Receiver Operating
Characteristic)-curves
○ Overview of validity coefficients relevant in personnel selection supplied so as to
guide the raters in formulating their judgement
○ Use of validity generalization is allowed
■ Only in case the cases are reasonable in similarity
○ Translation of the test → foreign studies may be generalized
■ Only if equivalence of the original version has been shown!
→ Deficiencies in the research design may lead to downwards adjustment of
the rating
Learning goals
1. What is validity?
a. Different forms of validity
b. How can you measure these forms?
2. How to test instruments on different forms of validity?
3. What can we do with the discrepancies that occur? (e.g. how you perceive
yourself, how others perceive you)
a. How can we explain these differences? (no straightforward answer)
Bron: Evers, A., Sijtsma, K., Lucassen, W., & Meijer, R. R. (2010)
Dutch Rating System for Test Quality
→ 7 criteria
1. Theoretical basis
2. Quality of the testing materials
3. Comprehensiveness of the manual
4. Norms
5. Reliability
6. Construct validity
7. Criterion validity
Validity
● Validity is the extent to which a test fulfils its purpose
● Differents forms of evidence on validity of tests should not be considered to represent
distinct types of validity
○ It’s a unitary concept
○ Important to collect evidence of validity that supports intended interpretation
● Standardized rating procedure is necessary to structure the concept of validity and
thereby the rating process
● Types of validity evidence that are construct-related are required for almost all tests,
whatever the purpose of the test use
● Types of validity evidence that are criterion-related will not be required for tests that
are not intended for prediction
Construct validity
● All research with respect to internal structure is considered relevant for construct validity
● Research concerning external structure is considered relevant for both construct and
criterion validity
● Evidence should support the claim: the test measures the intended trait or ability
○ So: it measures what it should measure
■ “What does the test measure?”
■ “Does the test measure the intended concept or does it partly or mainly
measure something else?”
, ● 6 types of research in support of construct validity
1. Research on the dimensionality of the item scores
Evidence based on factor analysis
● E.g. making measure for love
○ Items for love and liking
○ Fill out for loved one and friend
○ Factor analysis → you can see which items group together
■ In that way you can make categories
2. The psychometric quality of the items
3. Invariance of the factor structure and possible bias
→ supply evidence on internal structure (content)
4. Convergent and discriminant validity
5. Differences between relevant groups
6. Other research (e.g. research on criterion validity that is also relevant for
construct validity)
→ supply evidence on external structure
● Structure of items same as for reliability
○ First: provision of results is ascertained by means of a key item
○ Second: sufficiency of evidence supporting construct validity is assessed
○ Third: quality of research design is assessed
Criterion validity
● Research concerning external structure is considered relevant for both construct and
criterion validity
● Research on criterion-related evidence should demonstrate that a test score is a good
predictor of non-test behavior or outcome criteria
● Prediction can focus on
○ Past (retrospective validity)
○ Same moment in time (concurrent validity)
○ Future (predictive validity)
● Used in all tests → only: if test explicitly is not for prediction, than there is no need to
● Structure of items same as for construct validity
○ Recommendations contain a section on the use of ROC (Receiver Operating
Characteristic)-curves
○ Overview of validity coefficients relevant in personnel selection supplied so as to
guide the raters in formulating their judgement
○ Use of validity generalization is allowed
■ Only in case the cases are reasonable in similarity
○ Translation of the test → foreign studies may be generalized
■ Only if equivalence of the original version has been shown!
→ Deficiencies in the research design may lead to downwards adjustment of
the rating
