Test construction
Psychological measurements
Psychometrics: The science concerned with evaluating the attributes of
psychological tests (type of information, reliability and validity)
- Differential psychology: Study of individual differences (using an
average)
Psychological construct (unobservable): Theoretical concept of
psychological differences between individuals (made ‘observable’ trough
tests)
- Working memory, perceptual organization, extraversion,
neuroticism, etc
Psychological test (observable): Measurement instrument to quantify
individual differences in the construct
o Operational definitions/operationalizations
- Differences between psychological tests:
o Content
Depends on the construct to be measured
o Response required
Open-ended vs closed-ended
o Method of administration
Individual vs group
Interindividual differences: Comparing behavior of
different people
Intraindividual differences: Comparing behavior of
the same person at different points in time (or under
different circumstances)
o Use
Criterion referenced: Determine of someone passes a
certain cut-off
E.g., exam (between pass and fail)
Norm referenced: Compare score to population
E.g., intelligence test
o Timing
Speed test (time limit): Relatively easy questions, see
how far one comes
Power tests (no time limit): Different difficulty level, see
what one can solve
o The meaning of indicators
Formative versus reflective
Reflective measurement (most psychological
tests)
o Construct is assumed to cause differences in
the test scores
, o Item responses are indicators of the
construct
o Items are necessarily correlated
Higer values on the construct
increases scores on all items
Formative measurement
o The item response (indicators) defines the
construct
o Items are not necessarily correlated
Items independently contribute to the
construct
o E.g., Value of car (construct) is determined
by age, condition, size (items)
- Challenges in psychological measurement
o Complexity: Psychological phenomena are complex, high-
dimensional concepts
“You can’t reduce people to a number”
You need to carefully isolate a psychological dimension
(construct) and measure it
o Reactivity: People respond differently when they know that
they’re being measured
Demand characteristics: Changing behavior to
accommodate the researcher
Congruence: Ideal self doesn’t align with current
self
Social desirability: To impress the person doing the
measurements
o Observer bias: Expectations of the researcher may affect the
test
o Composite scores: Multiple item scores need to be combined
into one score
o Sensitivity: Unknown beforehand how sensitive a scale
should be
E.g., “How do you like this lecture until know?”
Too few responses categories: You may miss out on
individual differences
Too many response categories: People can’t
meaningfully distinguish the different categories
Measures ability to discriminate between meaningful
amounts of the dimension being measured
o Awareness: Many test administrators do not know about the
psychometric quality of the test
Evaluating the quality, being aware of the theories and
the limits of measurement tools
Dimensionality analysis
The dimensionality of a test is an important issue
o Dimensionality: The number of constructs a test measures
, - Unidimensional test: The test measures a single construct
E.g., D2: Test of attention, primary school arithmetic test
- Multidimensional test: The test measures multiple constructs
E.g., Big-5 personality inventory (OCEAN)
o With higher-order factor: A broader construct that
influences several specific traits or abilities
WAIS-IV intelligence test (e.g., verbal comprehension,
perceptual reasoning, working memory etc.)
Factor analysis: A statistical method to study the dimensionality of a test
- Confirmatory factor analysis (CFA): There is a clear theory about the
factor structure (the dimensionality of the constructs)
- Explanatory factor analysis (EFA): There is no theory about the factor
structure
o Step 1: Correlation matrix
EFA is commonly conducted on
the correlation matrix
Correlation matrix: A table
where you can see the correlations between different
items (helping see the dimensionality)
o Step 2: Check the Eigenvalues
As much Eigenvalues as items
Eigenvalue: Tells you
something about how many
variabilities this factor can account for in the data
o (High) variability/variance: How widely
the data is spread out from the mean
Numbers are always decreasing
o The higher numbers of variance are higher
factors (may be a construct)
Selecting a factor that might be a construct:
Kaiser criterion: Select all factor with an
Eigenvalue greater than
1.0 (don’t use), this will
cause a overestimation
of the number of factors
Scree plot
o Inflection point:
The point where
the item values stop decreasing (largely)
o Select all factors before that point
Psychological measurements
Psychometrics: The science concerned with evaluating the attributes of
psychological tests (type of information, reliability and validity)
- Differential psychology: Study of individual differences (using an
average)
Psychological construct (unobservable): Theoretical concept of
psychological differences between individuals (made ‘observable’ trough
tests)
- Working memory, perceptual organization, extraversion,
neuroticism, etc
Psychological test (observable): Measurement instrument to quantify
individual differences in the construct
o Operational definitions/operationalizations
- Differences between psychological tests:
o Content
Depends on the construct to be measured
o Response required
Open-ended vs closed-ended
o Method of administration
Individual vs group
Interindividual differences: Comparing behavior of
different people
Intraindividual differences: Comparing behavior of
the same person at different points in time (or under
different circumstances)
o Use
Criterion referenced: Determine of someone passes a
certain cut-off
E.g., exam (between pass and fail)
Norm referenced: Compare score to population
E.g., intelligence test
o Timing
Speed test (time limit): Relatively easy questions, see
how far one comes
Power tests (no time limit): Different difficulty level, see
what one can solve
o The meaning of indicators
Formative versus reflective
Reflective measurement (most psychological
tests)
o Construct is assumed to cause differences in
the test scores
, o Item responses are indicators of the
construct
o Items are necessarily correlated
Higer values on the construct
increases scores on all items
Formative measurement
o The item response (indicators) defines the
construct
o Items are not necessarily correlated
Items independently contribute to the
construct
o E.g., Value of car (construct) is determined
by age, condition, size (items)
- Challenges in psychological measurement
o Complexity: Psychological phenomena are complex, high-
dimensional concepts
“You can’t reduce people to a number”
You need to carefully isolate a psychological dimension
(construct) and measure it
o Reactivity: People respond differently when they know that
they’re being measured
Demand characteristics: Changing behavior to
accommodate the researcher
Congruence: Ideal self doesn’t align with current
self
Social desirability: To impress the person doing the
measurements
o Observer bias: Expectations of the researcher may affect the
test
o Composite scores: Multiple item scores need to be combined
into one score
o Sensitivity: Unknown beforehand how sensitive a scale
should be
E.g., “How do you like this lecture until know?”
Too few responses categories: You may miss out on
individual differences
Too many response categories: People can’t
meaningfully distinguish the different categories
Measures ability to discriminate between meaningful
amounts of the dimension being measured
o Awareness: Many test administrators do not know about the
psychometric quality of the test
Evaluating the quality, being aware of the theories and
the limits of measurement tools
Dimensionality analysis
The dimensionality of a test is an important issue
o Dimensionality: The number of constructs a test measures
, - Unidimensional test: The test measures a single construct
E.g., D2: Test of attention, primary school arithmetic test
- Multidimensional test: The test measures multiple constructs
E.g., Big-5 personality inventory (OCEAN)
o With higher-order factor: A broader construct that
influences several specific traits or abilities
WAIS-IV intelligence test (e.g., verbal comprehension,
perceptual reasoning, working memory etc.)
Factor analysis: A statistical method to study the dimensionality of a test
- Confirmatory factor analysis (CFA): There is a clear theory about the
factor structure (the dimensionality of the constructs)
- Explanatory factor analysis (EFA): There is no theory about the factor
structure
o Step 1: Correlation matrix
EFA is commonly conducted on
the correlation matrix
Correlation matrix: A table
where you can see the correlations between different
items (helping see the dimensionality)
o Step 2: Check the Eigenvalues
As much Eigenvalues as items
Eigenvalue: Tells you
something about how many
variabilities this factor can account for in the data
o (High) variability/variance: How widely
the data is spread out from the mean
Numbers are always decreasing
o The higher numbers of variance are higher
factors (may be a construct)
Selecting a factor that might be a construct:
Kaiser criterion: Select all factor with an
Eigenvalue greater than
1.0 (don’t use), this will
cause a overestimation
of the number of factors
Scree plot
o Inflection point:
The point where
the item values stop decreasing (largely)
o Select all factors before that point
