1
PSYCHOMETRICS – FURR (2022)
Inhoud
1. PSYCHOMETRICS AND THE IMPORTANCE OF PSYCHOLOGICAL MEASUREMENT..............................3
1.1. WHY PSYCHOLOGICAL TESTING MATTERS TO YOU....................................................................3
1.2. OBSERVABALE BEHAVIOR AND UNOBSERVABLE PSYCHOLOGICAL ATTRIBUTES........................3
1.3. PSYCHOLOGICAL TESTS: DEFINITION AND TYPES........................................................................4
1.4. WHAT IS PSYCHOMETRICS..........................................................................................................7
1.5. CHALLENGES TO MEASUREMENT IN PSYCHOLOGY....................................................................7
1.6. THE IMPORTANCE OF INDIVIDUAL DIFFERENCES.......................................................................8
1.7. BUT PSYCHOMETRICS GOES WELL BEYOND ‘DIFFERENTIAL’ PSYCHOLOGY................................9
2. SCALING...........................................................................................................................................10
2.1. FUNDAMENTAL ISSUES WITH NUMBERS..................................................................................10
2.2. UNITS OF MEASUREMENT........................................................................................................11
2.3. ADDITIVITY AND COUNTING.....................................................................................................12
2.4. FOUR SCALES OF MEASUREMENT.............................................................................................13
2.5. SCALES OF MEASUREMENT: PRACTICAL IMPLICATIONS...........................................................14
2.6. ADDITIONAL ISSUES REGARDING SCALES OF MEASUREMENT..................................................14
3. DIFFERENCES, CONSISTENCY, AND THE MEANING OF TEST SCORES...............................................15
3.1. THE NATURE OF VARIABILITY....................................................................................................15
3.2. IMPORTANCE OF INDIVIDUAL DIFFERENCES.............................................................................15
3.3. VARIABILITY AND DISTRIBUTIONS OF SCORES..........................................................................15
3.4. QUANTIFYING THE ASSOCIATION OR CONSISTENCY BETWEEN DISTRIBUTIONS......................16
3.5. VARIANCE AND COVARIANCE FOR “COMPOSITE VARIABLES”..................................................19
3.6. BINARY ITEMS...........................................................................................................................20
3.7. INTERPRETING TEST SCORES.....................................................................................................20
3.8. TEST NORMS.............................................................................................................................24
4. TEST DIMENSIONALITY AND FACTOR ANALYSIS...............................................................................26
4.1. TEST DIMENSIONALITY..............................................................................................................27
4.2. FACTOR ANALYSIS: EXAMINING THE DIMENSIONALITY OF A TEST...........................................29
5. RELIABILITY: conceptual basis..........................................................................................................37
5.1. OVERVIEW OF RELIABILITY AND CLASSICAL TEST THEORY........................................................37
5.2. OBSERVED SCORES, TRUE SCORES, AND MEASUREMENT ERROR............................................38
5.3. VARIANCES IN OBSERVED SCORES, TRUE SCORES AND ERROR SCORES...................................39
5.4. FOUR WAYS TO THINK OF RELIABILITY......................................................................................40
, 2
5.5. RELIABILITY AND THE STANDARD ERROR OF MEASUREMENT..................................................47
5.6. FROM THEORY TO PRACTICE: MEASUREMENT MODELS AND THEIR IMPLICATIONS FOR
ESTIMATING RELIABILITY.................................................................................................................48
6. EMPIRICAL ESTIMATES OF RELIABILITY............................................................................................54
6.1. ALTERNATE FORMS METHOD OF ESTIMATING RELIABILITY.....................................................54
6.2. TEST-RETEST METHOD OF ESTIMATING RELIABILITY................................................................55
6.3. INTERNAL CONSISTENCY METHOD OF ESTIMATING RELIABILITY..............................................57
6.4. SAMPLE HETEROGENEITY AND RELIABILITY GENERALIZATION.................................................65
6.5. RELIABILITY OF DIFFERENCE SCORES........................................................................................65
7. THE IMPORTANCE OF RELIABILITY...................................................................................................68
7.1. APPLIED BEHAVIORAL PRACTICE: EVALUATION OF AN INDIVIDUAL’S TEST SCORE..................68
7.2. BEHAVIORAL RESEARCH............................................................................................................69
7.3. TEST CONSTRUCTION AND REFINEMENT..................................................................................73
, 3
1.PSYCHOMETRICS AND THE IMPORTANCE OF
PSYCHOLOGICAL MEASUREMENT
Testing has robust implications – it should be done with the strongest tools and procedures
This book is about how to determine whether a test produces scores that are psychologically
meaningful and trustworthy. The principles and concepts (psychometrics) in this book are important
for creating tests that are psychologically meaningful and trustworthy.
1.1. WHY PSYCHOLOGICAL TESTING MATTERS TO YOU
Without a solid understanding of the basic principles of psychological measurement, test users risk
misinterpreting of misusing the information derived from psychological tests
If something is not measured or measured well, then it cannot be studied with any scientific validity
This book is about the important attributes of the instruments that psychologists use to measure
psychological attributes and processes
Fundamental questions this book addresses:
- What does it mean to attribute scores to characteristics such as intelligence, memory, self-
esteem, shyness, happiness or executive functioning?
- How do you know if a particular psychological measure is trustworthy and interpretable?
- How confident should you be when interpreting an individual’s score on a particular
psychological test?
- What kinds of questions should you ask to evaluate the quality of a psychological test?
- What are some of the different kinds of psychological measures?
- What are some of the challenges to psychological measurement?
- How is the measurement of psychological characteristics similar to and different from the
measurement of physical characteristics of objects?
- How should you interpret some of the technical information regarding psychological
measurement?
1.2. OBSERVABALE BEHAVIOR AND UNOBSERVABLE
PSYCHOLOGICAL ATTRIBUTES
Behavioral measurement is usually conducted for two purposes
- Interest in the behavior in its own right (e.g. facial expressions)
- As a way of assessing unobservable psychological attributes such as intelligence, depression,
knowledge, aptitude, extroversion or ability
Almost always, psychologists develop psychological tests as a way to sample the behavior that they
think reflects the underlying psychological attribute
1. Make an inference from an observable behavior to an unobservable psychological attribute
a. If this inference is reasonable, that interpretation of the behavior has a degree of
validity.
i. If the scores from a measure seem to be actually measuring the mental
state/process that we think they are measuring, then the interpretation of
scores on the measure is valid.
, 4
2. Measurement in psychology often, but not always, involves some type of theory linking a
psychological characteristic/process/state to an observable behavior that is thought to
reflect differences in that psychological attribute.
3. The psychological attribute can be a theoretical concept (hypothetical constructs or latent
variables).
a. They are theoretical psychological characteristics, attributes, processes or states that
cannot be directly observed (depression, self-esteem, intelligence, etc.)
b. The operations or procedures that we use to measure these hypothetical constructs
are called operational definitions.
1.3. Cronbach (1960): a PSYCHOLOGICAL
TESTS: systematic procedure DEFINITION AND
TYPES for comparing the
1.3.1. What behavior of two or more is a psychological
test?
Tests involve The behavioral samples The purpose of
behavioral must be collected in the test is to
samples of some systematic (clear detect differences
some kind and standardized) way between people
This definition is appealing, which is based on several important features:
- Generality
o Psychological tests come in many forms
o There’s many types of information produced by tests
- Tests must be capable of comparing the behavior of different people (interindividual
differences) or the behavior of the same individuals at different points in time or under
different circumstances (intraindividual differences).
o The purpose of measurement in psychology is to identify and, if possible, quantify
such interindividual or intraindividual differences.
o Inter- and intraindividual differences on test performance contribute to test score
variability, a necessary component of any attempt to measure any psychological
attribute.
1.3.2. Types of tests
Tests can vary in content:
- Achievement tests, aptitude tests, intelligence tests, personality tests, attitude surveys, etc.
Tests can vary with regard to the type of response required:
PSYCHOMETRICS – FURR (2022)
Inhoud
1. PSYCHOMETRICS AND THE IMPORTANCE OF PSYCHOLOGICAL MEASUREMENT..............................3
1.1. WHY PSYCHOLOGICAL TESTING MATTERS TO YOU....................................................................3
1.2. OBSERVABALE BEHAVIOR AND UNOBSERVABLE PSYCHOLOGICAL ATTRIBUTES........................3
1.3. PSYCHOLOGICAL TESTS: DEFINITION AND TYPES........................................................................4
1.4. WHAT IS PSYCHOMETRICS..........................................................................................................7
1.5. CHALLENGES TO MEASUREMENT IN PSYCHOLOGY....................................................................7
1.6. THE IMPORTANCE OF INDIVIDUAL DIFFERENCES.......................................................................8
1.7. BUT PSYCHOMETRICS GOES WELL BEYOND ‘DIFFERENTIAL’ PSYCHOLOGY................................9
2. SCALING...........................................................................................................................................10
2.1. FUNDAMENTAL ISSUES WITH NUMBERS..................................................................................10
2.2. UNITS OF MEASUREMENT........................................................................................................11
2.3. ADDITIVITY AND COUNTING.....................................................................................................12
2.4. FOUR SCALES OF MEASUREMENT.............................................................................................13
2.5. SCALES OF MEASUREMENT: PRACTICAL IMPLICATIONS...........................................................14
2.6. ADDITIONAL ISSUES REGARDING SCALES OF MEASUREMENT..................................................14
3. DIFFERENCES, CONSISTENCY, AND THE MEANING OF TEST SCORES...............................................15
3.1. THE NATURE OF VARIABILITY....................................................................................................15
3.2. IMPORTANCE OF INDIVIDUAL DIFFERENCES.............................................................................15
3.3. VARIABILITY AND DISTRIBUTIONS OF SCORES..........................................................................15
3.4. QUANTIFYING THE ASSOCIATION OR CONSISTENCY BETWEEN DISTRIBUTIONS......................16
3.5. VARIANCE AND COVARIANCE FOR “COMPOSITE VARIABLES”..................................................19
3.6. BINARY ITEMS...........................................................................................................................20
3.7. INTERPRETING TEST SCORES.....................................................................................................20
3.8. TEST NORMS.............................................................................................................................24
4. TEST DIMENSIONALITY AND FACTOR ANALYSIS...............................................................................26
4.1. TEST DIMENSIONALITY..............................................................................................................27
4.2. FACTOR ANALYSIS: EXAMINING THE DIMENSIONALITY OF A TEST...........................................29
5. RELIABILITY: conceptual basis..........................................................................................................37
5.1. OVERVIEW OF RELIABILITY AND CLASSICAL TEST THEORY........................................................37
5.2. OBSERVED SCORES, TRUE SCORES, AND MEASUREMENT ERROR............................................38
5.3. VARIANCES IN OBSERVED SCORES, TRUE SCORES AND ERROR SCORES...................................39
5.4. FOUR WAYS TO THINK OF RELIABILITY......................................................................................40
, 2
5.5. RELIABILITY AND THE STANDARD ERROR OF MEASUREMENT..................................................47
5.6. FROM THEORY TO PRACTICE: MEASUREMENT MODELS AND THEIR IMPLICATIONS FOR
ESTIMATING RELIABILITY.................................................................................................................48
6. EMPIRICAL ESTIMATES OF RELIABILITY............................................................................................54
6.1. ALTERNATE FORMS METHOD OF ESTIMATING RELIABILITY.....................................................54
6.2. TEST-RETEST METHOD OF ESTIMATING RELIABILITY................................................................55
6.3. INTERNAL CONSISTENCY METHOD OF ESTIMATING RELIABILITY..............................................57
6.4. SAMPLE HETEROGENEITY AND RELIABILITY GENERALIZATION.................................................65
6.5. RELIABILITY OF DIFFERENCE SCORES........................................................................................65
7. THE IMPORTANCE OF RELIABILITY...................................................................................................68
7.1. APPLIED BEHAVIORAL PRACTICE: EVALUATION OF AN INDIVIDUAL’S TEST SCORE..................68
7.2. BEHAVIORAL RESEARCH............................................................................................................69
7.3. TEST CONSTRUCTION AND REFINEMENT..................................................................................73
, 3
1.PSYCHOMETRICS AND THE IMPORTANCE OF
PSYCHOLOGICAL MEASUREMENT
Testing has robust implications – it should be done with the strongest tools and procedures
This book is about how to determine whether a test produces scores that are psychologically
meaningful and trustworthy. The principles and concepts (psychometrics) in this book are important
for creating tests that are psychologically meaningful and trustworthy.
1.1. WHY PSYCHOLOGICAL TESTING MATTERS TO YOU
Without a solid understanding of the basic principles of psychological measurement, test users risk
misinterpreting of misusing the information derived from psychological tests
If something is not measured or measured well, then it cannot be studied with any scientific validity
This book is about the important attributes of the instruments that psychologists use to measure
psychological attributes and processes
Fundamental questions this book addresses:
- What does it mean to attribute scores to characteristics such as intelligence, memory, self-
esteem, shyness, happiness or executive functioning?
- How do you know if a particular psychological measure is trustworthy and interpretable?
- How confident should you be when interpreting an individual’s score on a particular
psychological test?
- What kinds of questions should you ask to evaluate the quality of a psychological test?
- What are some of the different kinds of psychological measures?
- What are some of the challenges to psychological measurement?
- How is the measurement of psychological characteristics similar to and different from the
measurement of physical characteristics of objects?
- How should you interpret some of the technical information regarding psychological
measurement?
1.2. OBSERVABALE BEHAVIOR AND UNOBSERVABLE
PSYCHOLOGICAL ATTRIBUTES
Behavioral measurement is usually conducted for two purposes
- Interest in the behavior in its own right (e.g. facial expressions)
- As a way of assessing unobservable psychological attributes such as intelligence, depression,
knowledge, aptitude, extroversion or ability
Almost always, psychologists develop psychological tests as a way to sample the behavior that they
think reflects the underlying psychological attribute
1. Make an inference from an observable behavior to an unobservable psychological attribute
a. If this inference is reasonable, that interpretation of the behavior has a degree of
validity.
i. If the scores from a measure seem to be actually measuring the mental
state/process that we think they are measuring, then the interpretation of
scores on the measure is valid.
, 4
2. Measurement in psychology often, but not always, involves some type of theory linking a
psychological characteristic/process/state to an observable behavior that is thought to
reflect differences in that psychological attribute.
3. The psychological attribute can be a theoretical concept (hypothetical constructs or latent
variables).
a. They are theoretical psychological characteristics, attributes, processes or states that
cannot be directly observed (depression, self-esteem, intelligence, etc.)
b. The operations or procedures that we use to measure these hypothetical constructs
are called operational definitions.
1.3. Cronbach (1960): a PSYCHOLOGICAL
TESTS: systematic procedure DEFINITION AND
TYPES for comparing the
1.3.1. What behavior of two or more is a psychological
test?
Tests involve The behavioral samples The purpose of
behavioral must be collected in the test is to
samples of some systematic (clear detect differences
some kind and standardized) way between people
This definition is appealing, which is based on several important features:
- Generality
o Psychological tests come in many forms
o There’s many types of information produced by tests
- Tests must be capable of comparing the behavior of different people (interindividual
differences) or the behavior of the same individuals at different points in time or under
different circumstances (intraindividual differences).
o The purpose of measurement in psychology is to identify and, if possible, quantify
such interindividual or intraindividual differences.
o Inter- and intraindividual differences on test performance contribute to test score
variability, a necessary component of any attempt to measure any psychological
attribute.
1.3.2. Types of tests
Tests can vary in content:
- Achievement tests, aptitude tests, intelligence tests, personality tests, attitude surveys, etc.
Tests can vary with regard to the type of response required:
