Furr, & Bacharach (2014) – Psychometrics An introduction
Chapter 9 – Estimating and Evaluating Convergent and Discriminant
Validity Evidence
Psychological constructs are embedded in a theoretical context. Interconnections between
construct + other related constructs is called a nomological network. Suggests that a measure
of construct should be strongly associated with measures of some constructs but weakly
correlated with measures of other constructs.
Importance for validation: estimating degree to which test scores actually show the
predicted pattern of associations.
Methods for evaluating convergent and discriminant validity
- Focused associations: some measures have clear relevance for few specific variables
evaluating validity of interpretations for such measures. Research verifying the
predicted associations provides strong validity evidence.
o Method is to focus on few highly relevant criterion variables. Degree that test
scores are indeed correlated with those crucial variables = increased
confidence in test.
o Correlations are called validity coefficients, and are fundamental for
establishing validity. Large coefficient = increased confidence in test.
o Validity generalisation: process of evaluating a test’s validity coefficients across
large set of studies (e.g., SAT).
Usually small sample in research, due to limits on time, funding, and
other resources. BUT: small sample can have a drawback e.g., might
not be generalizable to other locations/populations.
Intended to evaluate predictive utility of test’s scores across range of
settings/times/situations etc.
Is a form of meta-analysis: combines results of several smaller
individual studies into 1 large analysis.
Can address 3 important issues:
Can reveal general level of predictive validity across all of
smaller individual studies.
Can reveal degree of variability among smaller individual
studies: lot of variability = not generalizable.
Can reveal the source of variability among studies: further
analyses might explain why results differ from each other, for
example due to differences in measurement of criterion
variable.
- Sets of correlations: correlations between test of interest + measures of criterion
variables make subjective judgement about degree to which correlations match
what would be expected (e.g., test about perfectionism, but also measures of several
psychological symptoms that are probably correlated with perfectionism, like OCD).
Then evaluate the correlation matrix. This is a common approach.
- Multitrait-multimethod matrices (MTMMM): obtain measures of several traits, each of
which is measured through several methods (e.g., self-report of social skill +
questionnaire on other traits, like impulsivity, emotional stability, etc.).
1
Chapter 9 – Estimating and Evaluating Convergent and Discriminant
Validity Evidence
Psychological constructs are embedded in a theoretical context. Interconnections between
construct + other related constructs is called a nomological network. Suggests that a measure
of construct should be strongly associated with measures of some constructs but weakly
correlated with measures of other constructs.
Importance for validation: estimating degree to which test scores actually show the
predicted pattern of associations.
Methods for evaluating convergent and discriminant validity
- Focused associations: some measures have clear relevance for few specific variables
evaluating validity of interpretations for such measures. Research verifying the
predicted associations provides strong validity evidence.
o Method is to focus on few highly relevant criterion variables. Degree that test
scores are indeed correlated with those crucial variables = increased
confidence in test.
o Correlations are called validity coefficients, and are fundamental for
establishing validity. Large coefficient = increased confidence in test.
o Validity generalisation: process of evaluating a test’s validity coefficients across
large set of studies (e.g., SAT).
Usually small sample in research, due to limits on time, funding, and
other resources. BUT: small sample can have a drawback e.g., might
not be generalizable to other locations/populations.
Intended to evaluate predictive utility of test’s scores across range of
settings/times/situations etc.
Is a form of meta-analysis: combines results of several smaller
individual studies into 1 large analysis.
Can address 3 important issues:
Can reveal general level of predictive validity across all of
smaller individual studies.
Can reveal degree of variability among smaller individual
studies: lot of variability = not generalizable.
Can reveal the source of variability among studies: further
analyses might explain why results differ from each other, for
example due to differences in measurement of criterion
variable.
- Sets of correlations: correlations between test of interest + measures of criterion
variables make subjective judgement about degree to which correlations match
what would be expected (e.g., test about perfectionism, but also measures of several
psychological symptoms that are probably correlated with perfectionism, like OCD).
Then evaluate the correlation matrix. This is a common approach.
- Multitrait-multimethod matrices (MTMMM): obtain measures of several traits, each of
which is measured through several methods (e.g., self-report of social skill +
questionnaire on other traits, like impulsivity, emotional stability, etc.).
1
