SUMMARY Measurement Theory and
Assessment I
Final Exam: Wednesday 28 March 2018
Index
à Introduction & Ethics 2
à Reliability
• Reliability I 4
• Reliability II 6
à Norms
• Norms I 10
• Norms II 12
à Validity 13
à Test construction 17
à Intelligence 21
à Introduction in Diagnostics 25
à Diagnostic errors 30
à Diagnostic Tools 33
CLAIMER
This summary is made by a student!
Studying from it and relying on it for 100% is your own responsibility.
THANKS & GOOD LUCK!! J
J YOU CAN DO IT !!!! CA
, 2
Introduction & Ethics
Diagnostic assessment
- Diagnosing – getting to know a situation thoroughly in order to be able to make a decision
- Psycho-diagnosing – research in the field of psychosocial functioning
o Reliable + valid description of psychosocial reality
o Find possible explanations for problems (don’t label but actually do label)
o Test these explanations
o Result of these tests à diagnosis
§ Diagnostic assessment is ideally repeatable
§ Diagnostic assessment ideally approaches reality
§ Inter-rate reliability psychiatric diagnosis with non-standardized tests (0-50%) and
with standardized tests (60-70%)
o Difficult
§ Cases are complex
§ Limited time
§ Confirmation bias – tendency to focus on confirming what you already believe
§ Availability heuristic – try to find a diagnosis that is more common
Tests
- Test – standardized procedure for taking a sample of behavior, described in categories or scores
- Formula classical test theory: X = T + e
o X = observed test score, T = true score, e = measurement error
Phases
1. Problem analysis
2. Classification and diagnosis
3. Treatment planning
4. Program/treatment evaluation
5. Self-knowledge (people who are very invested in a case (time/energy) are likely to see progress,
even when it has not occurred)
6. Scientific research
There are a lot of different types of psychological tests! As a psychologist, you have to...
- Know the psychometrical aspects of instruments
- Use these instruments in an ethical way
Example Gert – steps in a diagnostic procedure: Can ADHD explain the impulsive and hyperactive
behavior of Gert?
- Referral from GP (General Practitioner/huisarts/Hausarzt)
- Intake outpatient clinic ADHD
- Psychological/Pedagogical assessment
- Psychiatrical evaluation
Risks of testing
- Tests play an increasing role in society & are supposed to be objective
- Is a test valid? (does it measure what it aims to measure?)
- How & under what circumstances should a test be taken?
- Is a shortened version of a test reliable?
- How is the reference group determined?
, 3
Confounders
- Many psychological/pedagogical concepts are not perfectly defined
- Questions are misinterpreted
- Socially desirable answering
- Test assessment is not exactly as described in the manual (experimenters deviate from instructions)
Test properties
Standardization
1. Repeatability
o reliability + validity + norms
2. Integrality
o tests always contain a limited number of items
3. Scores
o the assessed psychological constructs are expressed by means of a number (e.g. score 1-5
on: “I slept well last week”)
4. Interpretation of scores Raw Score
o Compared to norm or standard score
o Often measured in no. of standard Comparison with Comparison with
deviations the score differs from cut-off point norm group
comparison
5. Prediction non-test behavior
o Test score predicts specific behavior, e.g. does IQ predict income?
o Replicating “normal” environment as much as possible
Assessment & Test Usage in the Netherlands
- NIP – Dutch Institute for Psychologists
- NVO – Dutch Institute for Pedagogic and Education Experts
à Quality assurance & professional ethics
- COTAN – Commissie Test Aangelegenheden Nederland
o Assess psychometrical properties & promote increase in the quality of tests
o Tasks: Inform test users about the quality of instruments & give creators of the instrument
feedback on the quality of their instruments
o COTAN sends tests to 2 professionals in the field who provide feedback on the test
o Criteria:
§ Principles of the test construction (measurement pretension)
§ Quality of test material and manual (standardization)
§ Norms, reliability, validity
2 instrument requirements
1. Psychodiagnostical instruments should meet the necessary psychometric criteria (COTAN)
2. Ethical use of psychodiagnostical instruments (NIP/NVO)
Ethics
Rules for assessment (NIP)
Goal: responsible, honest, respectful and competent
- Assessment must be relevant
- By authorized person
- Respect psychological and physical integrity
- Confidentiality
- Informed consent
- Independent and objective judgment
, 4
Ethics of publishers
- Psychometric characteristics must be up to date
- Only qualified people allowed to buy/use tests
- Digital use à always test on validity, reliability etc.
Reliability
Reliability I
Classical Test Theory (CTT)
Reliability <-> the theory of true and error scores
- Reliability – the correlation between the test scores of repeated assessments
o Number between 0 and 1
o The ratio between the X (observed test score) and T (true score)
o The consistency in measure
Measurement errors e.g. measuring IQ
- X > T à e is positive Observed score (X) = 110
- X < T à e is negative True score (T) = 100
- X will always be an estimate of T X > T à e is +10 (so it’s positive)
Sources of measurement error
1. Actual difference behavior is situational, so during the test the results can be
different than in an actual real-life situation
2. Item selection e.g. in depression questionnaire: it’s practically impossible to
assess all the aspects of depression in one questionnaire
3. Test administration e.g. experimenter effect (experimenter affects behavior),
environment, extraneous noises, distraction etc.
4. Test scoring open essay questions à how do you calculate grades/scores?
Always subjective idea from observer on right/wrong answer
5. Systematic errors (es) mistakes in the manual/test constructions
o Positive/negative
o Average measurement error is never zero, because it is the same (positive/negative) for
everyone (the number is not always exactly the same)
o If you are not sure if an error is systematic or not, see if everybody scores higher/lower, if
this is the case (so if there is a direction), you can safely assume it’s a systematic error
6. Unsystematic errors (eu)
o Can be positive and negative within the same population (e.g. someone has distraction
from noise outside, someone else is being “helped”/motivated)
o Average measurement is zero!
o Not related to the true score!
Basic principles of Classical Test Theory
- We are never sure of the true score, but we want to assess it à CTT gives us information about the
relation between X and T
- We will then be able to estimate the measurement error
- Then we can determine the Confidence Interval. Why? More certainty, is accurate representation
of the score
Assumptions
- CTT concerns only unsystematic measurement errors à systematic errors should be avoidable,
when you make the test it should actually be (close to) perfect, so systematic errors are impossible
Assessment I
Final Exam: Wednesday 28 March 2018
Index
à Introduction & Ethics 2
à Reliability
• Reliability I 4
• Reliability II 6
à Norms
• Norms I 10
• Norms II 12
à Validity 13
à Test construction 17
à Intelligence 21
à Introduction in Diagnostics 25
à Diagnostic errors 30
à Diagnostic Tools 33
CLAIMER
This summary is made by a student!
Studying from it and relying on it for 100% is your own responsibility.
THANKS & GOOD LUCK!! J
J YOU CAN DO IT !!!! CA
, 2
Introduction & Ethics
Diagnostic assessment
- Diagnosing – getting to know a situation thoroughly in order to be able to make a decision
- Psycho-diagnosing – research in the field of psychosocial functioning
o Reliable + valid description of psychosocial reality
o Find possible explanations for problems (don’t label but actually do label)
o Test these explanations
o Result of these tests à diagnosis
§ Diagnostic assessment is ideally repeatable
§ Diagnostic assessment ideally approaches reality
§ Inter-rate reliability psychiatric diagnosis with non-standardized tests (0-50%) and
with standardized tests (60-70%)
o Difficult
§ Cases are complex
§ Limited time
§ Confirmation bias – tendency to focus on confirming what you already believe
§ Availability heuristic – try to find a diagnosis that is more common
Tests
- Test – standardized procedure for taking a sample of behavior, described in categories or scores
- Formula classical test theory: X = T + e
o X = observed test score, T = true score, e = measurement error
Phases
1. Problem analysis
2. Classification and diagnosis
3. Treatment planning
4. Program/treatment evaluation
5. Self-knowledge (people who are very invested in a case (time/energy) are likely to see progress,
even when it has not occurred)
6. Scientific research
There are a lot of different types of psychological tests! As a psychologist, you have to...
- Know the psychometrical aspects of instruments
- Use these instruments in an ethical way
Example Gert – steps in a diagnostic procedure: Can ADHD explain the impulsive and hyperactive
behavior of Gert?
- Referral from GP (General Practitioner/huisarts/Hausarzt)
- Intake outpatient clinic ADHD
- Psychological/Pedagogical assessment
- Psychiatrical evaluation
Risks of testing
- Tests play an increasing role in society & are supposed to be objective
- Is a test valid? (does it measure what it aims to measure?)
- How & under what circumstances should a test be taken?
- Is a shortened version of a test reliable?
- How is the reference group determined?
, 3
Confounders
- Many psychological/pedagogical concepts are not perfectly defined
- Questions are misinterpreted
- Socially desirable answering
- Test assessment is not exactly as described in the manual (experimenters deviate from instructions)
Test properties
Standardization
1. Repeatability
o reliability + validity + norms
2. Integrality
o tests always contain a limited number of items
3. Scores
o the assessed psychological constructs are expressed by means of a number (e.g. score 1-5
on: “I slept well last week”)
4. Interpretation of scores Raw Score
o Compared to norm or standard score
o Often measured in no. of standard Comparison with Comparison with
deviations the score differs from cut-off point norm group
comparison
5. Prediction non-test behavior
o Test score predicts specific behavior, e.g. does IQ predict income?
o Replicating “normal” environment as much as possible
Assessment & Test Usage in the Netherlands
- NIP – Dutch Institute for Psychologists
- NVO – Dutch Institute for Pedagogic and Education Experts
à Quality assurance & professional ethics
- COTAN – Commissie Test Aangelegenheden Nederland
o Assess psychometrical properties & promote increase in the quality of tests
o Tasks: Inform test users about the quality of instruments & give creators of the instrument
feedback on the quality of their instruments
o COTAN sends tests to 2 professionals in the field who provide feedback on the test
o Criteria:
§ Principles of the test construction (measurement pretension)
§ Quality of test material and manual (standardization)
§ Norms, reliability, validity
2 instrument requirements
1. Psychodiagnostical instruments should meet the necessary psychometric criteria (COTAN)
2. Ethical use of psychodiagnostical instruments (NIP/NVO)
Ethics
Rules for assessment (NIP)
Goal: responsible, honest, respectful and competent
- Assessment must be relevant
- By authorized person
- Respect psychological and physical integrity
- Confidentiality
- Informed consent
- Independent and objective judgment
, 4
Ethics of publishers
- Psychometric characteristics must be up to date
- Only qualified people allowed to buy/use tests
- Digital use à always test on validity, reliability etc.
Reliability
Reliability I
Classical Test Theory (CTT)
Reliability <-> the theory of true and error scores
- Reliability – the correlation between the test scores of repeated assessments
o Number between 0 and 1
o The ratio between the X (observed test score) and T (true score)
o The consistency in measure
Measurement errors e.g. measuring IQ
- X > T à e is positive Observed score (X) = 110
- X < T à e is negative True score (T) = 100
- X will always be an estimate of T X > T à e is +10 (so it’s positive)
Sources of measurement error
1. Actual difference behavior is situational, so during the test the results can be
different than in an actual real-life situation
2. Item selection e.g. in depression questionnaire: it’s practically impossible to
assess all the aspects of depression in one questionnaire
3. Test administration e.g. experimenter effect (experimenter affects behavior),
environment, extraneous noises, distraction etc.
4. Test scoring open essay questions à how do you calculate grades/scores?
Always subjective idea from observer on right/wrong answer
5. Systematic errors (es) mistakes in the manual/test constructions
o Positive/negative
o Average measurement error is never zero, because it is the same (positive/negative) for
everyone (the number is not always exactly the same)
o If you are not sure if an error is systematic or not, see if everybody scores higher/lower, if
this is the case (so if there is a direction), you can safely assume it’s a systematic error
6. Unsystematic errors (eu)
o Can be positive and negative within the same population (e.g. someone has distraction
from noise outside, someone else is being “helped”/motivated)
o Average measurement is zero!
o Not related to the true score!
Basic principles of Classical Test Theory
- We are never sure of the true score, but we want to assess it à CTT gives us information about the
relation between X and T
- We will then be able to estimate the measurement error
- Then we can determine the Confidence Interval. Why? More certainty, is accurate representation
of the score
Assumptions
- CTT concerns only unsystematic measurement errors à systematic errors should be avoidable,
when you make the test it should actually be (close to) perfect, so systematic errors are impossible
