Test construction
Lecture 1
• Introduction
- Learning goals:
o To know and understand the principles of test and questionnaire
construction
o To know how tests and questionnaires for a particular aim and a
particular group are constructed
• Difference between test construction and test theory
- Test construction
o The administration and application of the test
▪ What does the test look like?
▪ Instructions for administration, scoring and application of the
tests
▪ The actual administrations of tests
• What information does it give?
• What is the usefulness of this information?
▪ How to combine scores?
- Test theory
o Statistical theory about the behaviour of item scores and test scores
o Examples: classical test theory and item response theory
o Important issues:
▪ Quantitative measures for the quality of items and tests for
target groups and respondents
- Both test theory and test construction are needed for the sensible use of tests
• Use of tests: in practice
- 1. Human resource management
o Personnel selection and development
- 2. Education: development and performance of students
o Identify deviating patterns of development in students
o Identify which type of school is the most suitable for a student
- 3. Psychodiagnostics
o Neuropsychology, clinical psychology, developmental psychology
- Judgments about individuals
• Use of tests: in research
- Testing of hypothesis, theory; theory building
- e.g., ‘Location and size of brain damage determines type and severity of
behavioural difficulties in the long term’
- Variables:
o indicators of location and size of brain damage
, o behavioural difficulties-e.g., Anxiety, Aggression, Childish behaviour,
Apathy, Lack of Insight
- Judgments about populations
• Definition
- A psychological or educational test is an instrument for the measurement of a
person’s maximum or typical performance under standardized conditions,
where the performance is assumed to reflect one or more latent attributes.
- Latent attributes: traits you cannot instantly see
• Type tests
- Typical performance test
o typifies person – no correct answers-
o e.g., personality, attitude, mental health
- Maximum performance test
o person’s achievement
▪ there are correct answers
▪ e.g., intelligence, ability level
▪ distinction:
• pure power test: no time limit (clock)
• time-limited power test: time is limited (fluency test)
• speed test: focus on time taken to solve problem (TMT)
• Standardization
- Test conditions are fixed
o e.g., test material, instructions, administration procedure, score
computing
- Aim: to ensure comparability of test performances between persons and test
occasions
- Difficult to achieve perfect standardization
- Specific aspects to standardize dependent on, for example, test or target
population
o For example: a test for children might need more flexible
administration, such as giving breaks in between, than tests for adults
• Latent attribute
- Attribute that cannot be measured directly
o e.g., verbal ability, arithmetic skills, severity of depression
- Test score (S) should reflect the latent attribute of interest (T; true score)
o causal relationship between attribute and test score
o thus: if two persons differ on the attribute, the test scores differ as
well, and the other way around
o T: true score is the true representation, but becomes a test score (S)
because of error
o T → S measurement error
, • Some important terminology
- Item
o smallest test unit, on which person is scored
o score can be the same as person’s response
- Subtest (also denoted as subscale, or just scale)
o independent part of a test
o indicative of an attribute
o consists of various items
How is a test constructed???
• …and 6. Test use
- Steps 1-5 are about the construction of a test (measurement device):
Important !
- In practice, psychologists use tests for e.g., diagnostics, and prediction
- A test can be reliable and valid test but at the end it is the decision that
counts
- Because it is so important in practice we start with
- HOW TO USE TEST SCORES
• Judgment on the basis of test scores
- A mind-blowing experimental finding:
- Setting: Admission testing (Sarbin, 1943)
- Two conditions
o (1) Prediction of college grades on the basis of high school grades and
an admission test
o (2) Prediction of college grades on the basis of these test scores and
extra information from interviewing the candidates
- Result: The prediction on the basis of only combining the two test scores
resulted in a correlation of r = .45 with college grades
, - Adding the information from the interviews resulted in r = .35 with college
grades
- Uhhhh?? More information from interviews results in lower prediction??
- Yes, sir !
o People add error→ so a prediction based on more objective,
standardised measures such as performance tests will result in more
accurate predictions than a more subjective measure, such as an
interview
• Meehl (1954) judgment: two basic options (article to read on Brightspace)
- “mechanical or statistical prediction: x1+ x2+ x3 = xt2.
- Expert (“clinical”) judgment: Expert combines information “in the head”
- Many studies found that statistical methods outperform human judgment for
combining multiple data features
• Why is that?
- People do not minimize error, but add error
- Error that feels great: As long as people can make coherent story about
something they will accept that story no matter any empirical evidence
- Experts are inconsistent in combining data
- Experts judgment can often be replaced by simple rules
- It is very difficult to improve decisions adding all kinds of interaction terms
• Realize what this means ..
- An expert (e.g., admission officer, clinical psychologist) who interprets test
scores, thinks about it and then makes a decision is often less accurate (less
valid) than information from just adding test scores
- Golden rule of prediction: take a number of valid “test” scores (can also be
interview scores) and add them up !
• Message
- So these findings illustrate that
o 1. When combining information use a rule
o 2. Test scores should be valid
- In this course we talk about test construction, that is how a good test should
be constructed
- The assignment that we got illustrates how difficult it is to beat formulas and
that we should be very careful to trust our judgments and predictions using
test scores combined “in the head”
• How can we improve decision making using test scores?
- Use a rule, because “random fluctuations” are reduced to zero (we are
perfectly consistent, reliable in our judgments)
- Use independent judgments and take the mean (or median)
o The errors and biases of individual guesses are cancelled out when
looking at the mean of independent guesses
- Famous experiment Galton: Wisdom of the crowd
Lecture 1
• Introduction
- Learning goals:
o To know and understand the principles of test and questionnaire
construction
o To know how tests and questionnaires for a particular aim and a
particular group are constructed
• Difference between test construction and test theory
- Test construction
o The administration and application of the test
▪ What does the test look like?
▪ Instructions for administration, scoring and application of the
tests
▪ The actual administrations of tests
• What information does it give?
• What is the usefulness of this information?
▪ How to combine scores?
- Test theory
o Statistical theory about the behaviour of item scores and test scores
o Examples: classical test theory and item response theory
o Important issues:
▪ Quantitative measures for the quality of items and tests for
target groups and respondents
- Both test theory and test construction are needed for the sensible use of tests
• Use of tests: in practice
- 1. Human resource management
o Personnel selection and development
- 2. Education: development and performance of students
o Identify deviating patterns of development in students
o Identify which type of school is the most suitable for a student
- 3. Psychodiagnostics
o Neuropsychology, clinical psychology, developmental psychology
- Judgments about individuals
• Use of tests: in research
- Testing of hypothesis, theory; theory building
- e.g., ‘Location and size of brain damage determines type and severity of
behavioural difficulties in the long term’
- Variables:
o indicators of location and size of brain damage
, o behavioural difficulties-e.g., Anxiety, Aggression, Childish behaviour,
Apathy, Lack of Insight
- Judgments about populations
• Definition
- A psychological or educational test is an instrument for the measurement of a
person’s maximum or typical performance under standardized conditions,
where the performance is assumed to reflect one or more latent attributes.
- Latent attributes: traits you cannot instantly see
• Type tests
- Typical performance test
o typifies person – no correct answers-
o e.g., personality, attitude, mental health
- Maximum performance test
o person’s achievement
▪ there are correct answers
▪ e.g., intelligence, ability level
▪ distinction:
• pure power test: no time limit (clock)
• time-limited power test: time is limited (fluency test)
• speed test: focus on time taken to solve problem (TMT)
• Standardization
- Test conditions are fixed
o e.g., test material, instructions, administration procedure, score
computing
- Aim: to ensure comparability of test performances between persons and test
occasions
- Difficult to achieve perfect standardization
- Specific aspects to standardize dependent on, for example, test or target
population
o For example: a test for children might need more flexible
administration, such as giving breaks in between, than tests for adults
• Latent attribute
- Attribute that cannot be measured directly
o e.g., verbal ability, arithmetic skills, severity of depression
- Test score (S) should reflect the latent attribute of interest (T; true score)
o causal relationship between attribute and test score
o thus: if two persons differ on the attribute, the test scores differ as
well, and the other way around
o T: true score is the true representation, but becomes a test score (S)
because of error
o T → S measurement error
, • Some important terminology
- Item
o smallest test unit, on which person is scored
o score can be the same as person’s response
- Subtest (also denoted as subscale, or just scale)
o independent part of a test
o indicative of an attribute
o consists of various items
How is a test constructed???
• …and 6. Test use
- Steps 1-5 are about the construction of a test (measurement device):
Important !
- In practice, psychologists use tests for e.g., diagnostics, and prediction
- A test can be reliable and valid test but at the end it is the decision that
counts
- Because it is so important in practice we start with
- HOW TO USE TEST SCORES
• Judgment on the basis of test scores
- A mind-blowing experimental finding:
- Setting: Admission testing (Sarbin, 1943)
- Two conditions
o (1) Prediction of college grades on the basis of high school grades and
an admission test
o (2) Prediction of college grades on the basis of these test scores and
extra information from interviewing the candidates
- Result: The prediction on the basis of only combining the two test scores
resulted in a correlation of r = .45 with college grades
, - Adding the information from the interviews resulted in r = .35 with college
grades
- Uhhhh?? More information from interviews results in lower prediction??
- Yes, sir !
o People add error→ so a prediction based on more objective,
standardised measures such as performance tests will result in more
accurate predictions than a more subjective measure, such as an
interview
• Meehl (1954) judgment: two basic options (article to read on Brightspace)
- “mechanical or statistical prediction: x1+ x2+ x3 = xt2.
- Expert (“clinical”) judgment: Expert combines information “in the head”
- Many studies found that statistical methods outperform human judgment for
combining multiple data features
• Why is that?
- People do not minimize error, but add error
- Error that feels great: As long as people can make coherent story about
something they will accept that story no matter any empirical evidence
- Experts are inconsistent in combining data
- Experts judgment can often be replaced by simple rules
- It is very difficult to improve decisions adding all kinds of interaction terms
• Realize what this means ..
- An expert (e.g., admission officer, clinical psychologist) who interprets test
scores, thinks about it and then makes a decision is often less accurate (less
valid) than information from just adding test scores
- Golden rule of prediction: take a number of valid “test” scores (can also be
interview scores) and add them up !
• Message
- So these findings illustrate that
o 1. When combining information use a rule
o 2. Test scores should be valid
- In this course we talk about test construction, that is how a good test should
be constructed
- The assignment that we got illustrates how difficult it is to beat formulas and
that we should be very careful to trust our judgments and predictions using
test scores combined “in the head”
• How can we improve decision making using test scores?
- Use a rule, because “random fluctuations” are reduced to zero (we are
perfectly consistent, reliable in our judgments)
- Use independent judgments and take the mean (or median)
o The errors and biases of individual guesses are cancelled out when
looking at the mean of independent guesses
- Famous experiment Galton: Wisdom of the crowd
