1
Lecture notes Test construction (PSMM-6)
Lecture 1: Developing maximum and typical performance tests
Learning goals:
- To know and understand the principles of test and questionnaire construction
- To know how tests and questionnaires for a particular aim and a particular group are
effectively constructed, evaluated and interpreted
Psychological and educational tests:
Test construction - development and Test theory – statistical theory about
application behavior of item scores and test scores
- What does the test look like? - Examples: Classical test theory,
- Instructions for administration, item response theory
scoring and interpretation - Important issues: quantitative
- Actual administrations of tests (what measures for the quality of items
information does it give? what is the and tests for target groups of
usefulness of this information, and respondents
for whom (individuals, policy)?)
Both are needed for a sensible use of tests.
Use of tests:
In practice In research
1. Human Resource management – - Testing of hypothesis, theory, theory
Personnel selection and building
development - E.g., ‘Location and size of brain
2. Education: individual development damage determines type and
and performance of students – severity of behavioural difficulties in
Identify deviating patterns of the long term’
development – Prediction of most Judgments on populations
suitable type of high school
3. Psychodiagnostics –
Neuropsychology, clinical
psychology, developmental
psychology
Judgments on individuals
Definition: A psychological or educational test is an instrument for the measurement of a
person’s maximum or typical performance under standardized conditions, where the
performance is assumed to reflect one or more latent attributes.
Test types:
Typical performance test= typifies person, Maximum performance test= person’s
no correct answers achievement
- Personality - Intelligence
, 2
- Attitude - Ability level
- Mental health
Standardization;
- Test conditions are fixed (e.g., test material, instructions, administration procedure,
score computing)
- Aim: to ensure comparability of test performances between persons and test
occasions
- Difficult to achieve perfect standardization
- Specific aspects to standardize dependent on, for example, test or target population
Latent attribute - Attribute that cannot be measured directly (e.g., verbal
ability, arithmetic skills, severity of depression)
- Test score (X) should reflect the latent attribute of
interest (T; true score)
● Causal relationship between attribute and test
score
● Thus: if two persons differ on the attribute, the
test scores differ as well, and the other way
around
T= true score (a latent attribute because it cannot be observed
directly)
X= observed score (therefore placed in a square because it can
be observed directly)
Arrows= causal directions (we assume that the true score has a
causal effect on the observed score).
Item - Smallest test unit, on which person is scored
- Score can be the same as persons response
Subtest (subscale) - Independent part of a test
- Indicative of an attribute
- Consists of various items
Test construction:
1. Define construct:
● Constructs > abstract, theoretical concepts
● Literature search
, 3
● Dimensionality: number of latent attributes (variables), which effects test
performance + unidimensional vs multidimensional test
2. Develop the test:
● Measurement mode of the test: self-performance mode, self-evaluation mode,
other-evaluation mode.
● Objectives of the test:
- Research versus practice
- Individual or group level
- Description versus diagnosis versus decision-making
1. To describe performances (description)
2. To add a conclusion to a description (diagnosis)
3. To make decisions based on tests (decision-making)
● Population and subpopulations of testees
- Be as specific as possible
- Inclusion and exclusion criteria
- Too broad → implications for norm groups and their
representativeness
● Administration mode: oral, paper-and-pencil, computerized, computerized
adaptive test administration
● Conceptual framework of the test:
- More specific than just definition; it helps to write items
- Typical performance: three broad classes of strategies
1. Intuitive class - relation between construct and items is of an
intuitive nature
2. Deductive class - start from theoretical or conceptual notions of
the construct
3. Inductive class - constructs to be measured cannot be defined
beforehand, but are identified using empirical data (i.e.,
association measures)
● Response mode: many, see book. Frequently-used scales:
- Dichotomous = binary (yes/no, true/false, correct/incorrect - typically
coded as 0, 1)
- Ordinal polytomous (never/sometimes/often)
● Item writing: book describes different concrete guidelines (both for typical and
maximum performance test items). In general:
- Each item represents one idea
- Be specific
- Use positive and negative formulated items
- Avoid expressions and jargon
- Consider the reading level of the user
- Avoid the use of ‘not’
3. Pilot study:
● Check whether instructions and items are clear
● Three types of studies
1. Experts’ pilot - concept items are reviewed by experts
2. Test takers’ pilot - concept items are administered to small group test
takers from the target population (use read-aloud protocol or think out
Lecture notes Test construction (PSMM-6)
Lecture 1: Developing maximum and typical performance tests
Learning goals:
- To know and understand the principles of test and questionnaire construction
- To know how tests and questionnaires for a particular aim and a particular group are
effectively constructed, evaluated and interpreted
Psychological and educational tests:
Test construction - development and Test theory – statistical theory about
application behavior of item scores and test scores
- What does the test look like? - Examples: Classical test theory,
- Instructions for administration, item response theory
scoring and interpretation - Important issues: quantitative
- Actual administrations of tests (what measures for the quality of items
information does it give? what is the and tests for target groups of
usefulness of this information, and respondents
for whom (individuals, policy)?)
Both are needed for a sensible use of tests.
Use of tests:
In practice In research
1. Human Resource management – - Testing of hypothesis, theory, theory
Personnel selection and building
development - E.g., ‘Location and size of brain
2. Education: individual development damage determines type and
and performance of students – severity of behavioural difficulties in
Identify deviating patterns of the long term’
development – Prediction of most Judgments on populations
suitable type of high school
3. Psychodiagnostics –
Neuropsychology, clinical
psychology, developmental
psychology
Judgments on individuals
Definition: A psychological or educational test is an instrument for the measurement of a
person’s maximum or typical performance under standardized conditions, where the
performance is assumed to reflect one or more latent attributes.
Test types:
Typical performance test= typifies person, Maximum performance test= person’s
no correct answers achievement
- Personality - Intelligence
, 2
- Attitude - Ability level
- Mental health
Standardization;
- Test conditions are fixed (e.g., test material, instructions, administration procedure,
score computing)
- Aim: to ensure comparability of test performances between persons and test
occasions
- Difficult to achieve perfect standardization
- Specific aspects to standardize dependent on, for example, test or target population
Latent attribute - Attribute that cannot be measured directly (e.g., verbal
ability, arithmetic skills, severity of depression)
- Test score (X) should reflect the latent attribute of
interest (T; true score)
● Causal relationship between attribute and test
score
● Thus: if two persons differ on the attribute, the
test scores differ as well, and the other way
around
T= true score (a latent attribute because it cannot be observed
directly)
X= observed score (therefore placed in a square because it can
be observed directly)
Arrows= causal directions (we assume that the true score has a
causal effect on the observed score).
Item - Smallest test unit, on which person is scored
- Score can be the same as persons response
Subtest (subscale) - Independent part of a test
- Indicative of an attribute
- Consists of various items
Test construction:
1. Define construct:
● Constructs > abstract, theoretical concepts
● Literature search
, 3
● Dimensionality: number of latent attributes (variables), which effects test
performance + unidimensional vs multidimensional test
2. Develop the test:
● Measurement mode of the test: self-performance mode, self-evaluation mode,
other-evaluation mode.
● Objectives of the test:
- Research versus practice
- Individual or group level
- Description versus diagnosis versus decision-making
1. To describe performances (description)
2. To add a conclusion to a description (diagnosis)
3. To make decisions based on tests (decision-making)
● Population and subpopulations of testees
- Be as specific as possible
- Inclusion and exclusion criteria
- Too broad → implications for norm groups and their
representativeness
● Administration mode: oral, paper-and-pencil, computerized, computerized
adaptive test administration
● Conceptual framework of the test:
- More specific than just definition; it helps to write items
- Typical performance: three broad classes of strategies
1. Intuitive class - relation between construct and items is of an
intuitive nature
2. Deductive class - start from theoretical or conceptual notions of
the construct
3. Inductive class - constructs to be measured cannot be defined
beforehand, but are identified using empirical data (i.e.,
association measures)
● Response mode: many, see book. Frequently-used scales:
- Dichotomous = binary (yes/no, true/false, correct/incorrect - typically
coded as 0, 1)
- Ordinal polytomous (never/sometimes/often)
● Item writing: book describes different concrete guidelines (both for typical and
maximum performance test items). In general:
- Each item represents one idea
- Be specific
- Use positive and negative formulated items
- Avoid expressions and jargon
- Consider the reading level of the user
- Avoid the use of ‘not’
3. Pilot study:
● Check whether instructions and items are clear
● Three types of studies
1. Experts’ pilot - concept items are reviewed by experts
2. Test takers’ pilot - concept items are administered to small group test
takers from the target population (use read-aloud protocol or think out
