INTRODUCTION & ETHICS
Learning objectives:
- I understand the importance of diagnostic tests.
- I understand which aspects define the quality of diagnostic instruments.
- I know how to administer, interpret, and communicate a test ethically.
Diagnostics: getting to know a situation through and through to be able to make a decision
- Should ideally be (1) repeatable, and (2) approaches reality
Psychodiagnostics: getting to know an individual’s psychosocial functioning
- Reliable and valid description of psychosocial reality
- Find explanations for problems
- Test explanations
Disadvantages:
- Difficult, complex constructs
- Limited time
- Confirmation bias: tendency to interpret new evidence as confirmation of one's existing
beliefs or theories
- Availability heuristic: tendency to check for symptoms that are related to disorders with a high
prevalence
TESTS: standardised procedure for taking a sample of behaviour, described in categories or scores
1. Problem analysis
2. Classification and diagnosis
3. Treatment planning
4. program/treatment evaluation
5. Self-knowledge
Questions to consider in tests:
- Does it measure what it aims to measure? (ie validity)
- How and under what circumstances should you test?
- Is a short version of a test (just as) reliable?
- How is the reference group determined?
Confounders: measurement errors
Eg - Ill-defined constructs
- Misinterpretation of questions
- Social desirability answering/context
- Negligent use of manual
Test properties: standardisation
1. Repeatability: assess the same score provided that construct to be measured remains the same
(reliability)
2. Sample of behaviour/integrality: limited number of items
3. Use of scores or categories: express constructs using numbers
4. Use of norm or standard to classify: comparison of raw score to a cut-off value or norm group
5. Prediction of non-test behaviour: test score predicts specific behaviour (validity)
,Instrument requirement
(1) psychometric criteria (eg Committee on Tests and Testing in the Netherlands or COTAN)
- inform users about quality of instruments
- provide feedback to developers on quality of instruments
COTAN criteria
(a) Principles of the test construction (ie goal, group, function)
(b) Standardisation (ie quality of test material and manual)
(c) Norms (ie representative reference group)
(d) Reliability (ie consistency/repeatability of score)
(e) Validity (ie assessing the right construct)
(2) ethical use (eg Guidelines for use of tests/NIP)
Goal of Code of Ethics: responsibility, integrity, respect and expertise
- Role integrity
- Confidentiality
- Informed consent
- Independent and objective (ie reporting without jargon)
,RELIABILITY
Learning objective:
- I know the definition of reliability according to classic test theory, and I can adequately calculate and
interpret it.
- I understand the different methods of assessing reliability, and I know which is appropriate for a
specific case.
- I know what the standard error of measurement is.
- I know how to determine the confidence intervals of scores and difference-scores.
Methods: a systematic approach to how you should conduct (empirical) research
Statistics: the tools you need to conduct empirical research
Measurement Theory and Assessment: implications of research results for (individual) test scores
Reliability: the correlation between test scores after repeated assessments or between items within the test
consistency/reproducibility of scores
- Consistency/reproducibility of scores (the smaller the noise of measurement error, the better)
- Score between 0 and 1; where 0 is no correlation, and 1 is a perfect correlation
Sources of errors
● (Actual difference)
○ behaviour is situational
○ extenuating circumstances
● Measurement errors
○ Item selection (sample of behaviour)
○ Test administration (examinee/examiner/environment)
○ Test scoring (eg multiple-choice vs essay questions)
○ Systematic errors of measurement
Systematic and unsystematic errors of measurement
Unsystematic errors = eu Systematic errors = ea
- Both positive and negative - Either positive or negative
- Average measurement error = 0 - Average measurement error ≠ 0
- Unpredictable errors: errors during test - Predictable errors: error in test construction
assessment resulting in inconsistency that is consistent
- Measure of reliability (introduces noise in - Measure of validity (consistent score is
test score; lower the noise, more reliable the possible, but might not be valid)
test)
, Classic Test Theory: the theory of true and error scores
- Measures similarity between true score (T) and assessed score (X)
- Estimate unsystematic errors (e)
- Determines confidence interval (CI); a more accurate representation of score
- The smaller the CI, the less certain you can be about it including the actual true score
X=T+e
X = observed test score
T = true score (to be assessed); consistency of construct
e = measurement error (factors unrelated to true score); inconsistency of errors
- Test score X is always an estimation of true score T
Assumptions:
● If you assess a large enough sample (>500), the average will be the true score of the individual
● CTT is about unsystemtaic measurement errors
○ Because systematic errors are ‘avoidable’
● Characteristics unsystematic errors
○ Measurement errors are positive and negative
■ Average emasurement error = 0
○ Measurement errors are not related to T
○ Measurement errors are not related to each other
○ Measurement errors are normally distributed
■ Measurement error → variability of obtained scores
■ Variability of obtained scores → variance in statistics
Measurement errors are normally distributed (in a large enough sample)
- Percentage within segments of normal distribution
- Deviation from the average expressed in standard deviations
Test score
● Average measurement error e = 0
● Average test score X (over many assessments) = average true score T
Variability of test score
● Variance test scores X = variance true scores T + variance measurement errors
○ Note: variance measurement errors ≠ 0
σx² = σT² + σe²
σx² = variance test score X
σT² = variance true score T
σe² = variance measurement error e
Learning objectives:
- I understand the importance of diagnostic tests.
- I understand which aspects define the quality of diagnostic instruments.
- I know how to administer, interpret, and communicate a test ethically.
Diagnostics: getting to know a situation through and through to be able to make a decision
- Should ideally be (1) repeatable, and (2) approaches reality
Psychodiagnostics: getting to know an individual’s psychosocial functioning
- Reliable and valid description of psychosocial reality
- Find explanations for problems
- Test explanations
Disadvantages:
- Difficult, complex constructs
- Limited time
- Confirmation bias: tendency to interpret new evidence as confirmation of one's existing
beliefs or theories
- Availability heuristic: tendency to check for symptoms that are related to disorders with a high
prevalence
TESTS: standardised procedure for taking a sample of behaviour, described in categories or scores
1. Problem analysis
2. Classification and diagnosis
3. Treatment planning
4. program/treatment evaluation
5. Self-knowledge
Questions to consider in tests:
- Does it measure what it aims to measure? (ie validity)
- How and under what circumstances should you test?
- Is a short version of a test (just as) reliable?
- How is the reference group determined?
Confounders: measurement errors
Eg - Ill-defined constructs
- Misinterpretation of questions
- Social desirability answering/context
- Negligent use of manual
Test properties: standardisation
1. Repeatability: assess the same score provided that construct to be measured remains the same
(reliability)
2. Sample of behaviour/integrality: limited number of items
3. Use of scores or categories: express constructs using numbers
4. Use of norm or standard to classify: comparison of raw score to a cut-off value or norm group
5. Prediction of non-test behaviour: test score predicts specific behaviour (validity)
,Instrument requirement
(1) psychometric criteria (eg Committee on Tests and Testing in the Netherlands or COTAN)
- inform users about quality of instruments
- provide feedback to developers on quality of instruments
COTAN criteria
(a) Principles of the test construction (ie goal, group, function)
(b) Standardisation (ie quality of test material and manual)
(c) Norms (ie representative reference group)
(d) Reliability (ie consistency/repeatability of score)
(e) Validity (ie assessing the right construct)
(2) ethical use (eg Guidelines for use of tests/NIP)
Goal of Code of Ethics: responsibility, integrity, respect and expertise
- Role integrity
- Confidentiality
- Informed consent
- Independent and objective (ie reporting without jargon)
,RELIABILITY
Learning objective:
- I know the definition of reliability according to classic test theory, and I can adequately calculate and
interpret it.
- I understand the different methods of assessing reliability, and I know which is appropriate for a
specific case.
- I know what the standard error of measurement is.
- I know how to determine the confidence intervals of scores and difference-scores.
Methods: a systematic approach to how you should conduct (empirical) research
Statistics: the tools you need to conduct empirical research
Measurement Theory and Assessment: implications of research results for (individual) test scores
Reliability: the correlation between test scores after repeated assessments or between items within the test
consistency/reproducibility of scores
- Consistency/reproducibility of scores (the smaller the noise of measurement error, the better)
- Score between 0 and 1; where 0 is no correlation, and 1 is a perfect correlation
Sources of errors
● (Actual difference)
○ behaviour is situational
○ extenuating circumstances
● Measurement errors
○ Item selection (sample of behaviour)
○ Test administration (examinee/examiner/environment)
○ Test scoring (eg multiple-choice vs essay questions)
○ Systematic errors of measurement
Systematic and unsystematic errors of measurement
Unsystematic errors = eu Systematic errors = ea
- Both positive and negative - Either positive or negative
- Average measurement error = 0 - Average measurement error ≠ 0
- Unpredictable errors: errors during test - Predictable errors: error in test construction
assessment resulting in inconsistency that is consistent
- Measure of reliability (introduces noise in - Measure of validity (consistent score is
test score; lower the noise, more reliable the possible, but might not be valid)
test)
, Classic Test Theory: the theory of true and error scores
- Measures similarity between true score (T) and assessed score (X)
- Estimate unsystematic errors (e)
- Determines confidence interval (CI); a more accurate representation of score
- The smaller the CI, the less certain you can be about it including the actual true score
X=T+e
X = observed test score
T = true score (to be assessed); consistency of construct
e = measurement error (factors unrelated to true score); inconsistency of errors
- Test score X is always an estimation of true score T
Assumptions:
● If you assess a large enough sample (>500), the average will be the true score of the individual
● CTT is about unsystemtaic measurement errors
○ Because systematic errors are ‘avoidable’
● Characteristics unsystematic errors
○ Measurement errors are positive and negative
■ Average emasurement error = 0
○ Measurement errors are not related to T
○ Measurement errors are not related to each other
○ Measurement errors are normally distributed
■ Measurement error → variability of obtained scores
■ Variability of obtained scores → variance in statistics
Measurement errors are normally distributed (in a large enough sample)
- Percentage within segments of normal distribution
- Deviation from the average expressed in standard deviations
Test score
● Average measurement error e = 0
● Average test score X (over many assessments) = average true score T
Variability of test score
● Variance test scores X = variance true scores T + variance measurement errors
○ Note: variance measurement errors ≠ 0
σx² = σT² + σe²
σx² = variance test score X
σT² = variance true score T
σe² = variance measurement error e
