Construction and Analysis of Questionnaires
Aantekeningen
Blok 3, 2024
Tilburg University
,Inhoudsopgave
Construction and Analysis of Questionnaires ................................................................................................... 1
Prelude............................................................................................................................................................. 4
Lecture 1: Why Questionnaires? ...................................................................................................................... 5
Definition of a Questionnaire ........................................................................................................................ 5
What is an ‘Item’? .......................................................................................................................................... 6
What is the purpose of a questionnaire? ...................................................................................................... 6
Measurement in technical sciences ............................................................................................................ 10
Measurement in social sciences .................................................................................................................. 10
Validity ......................................................................................................................................................... 10
Reliability ..................................................................................................................................................... 10
Content of questionnaires ........................................................................................................................... 11
Modes and question types .......................................................................................................................... 11
Former exam questions ............................................................................................................................... 11
Lecture 2: Making constructs measurable ...................................................................................................... 12
What are constructs? .................................................................................................................................. 12
Making constructs measurable ................................................................................................................... 13
Six established methods of Scale Construction ........................................................................................... 14
1. Construct Method .............................................................................................................................. 15
2. Facet Design Method ......................................................................................................................... 17
3. Rational Method ................................................................................................................................ 18
4. Prototypical Method .......................................................................................................................... 19
5. Internal Method ................................................................................................................................. 19
6. External Method ................................................................................................................................ 20
Exam questions............................................................................................................................................ 21
Lecture 3: Questionnaires in Practice ............................................................................................................. 23
Formulating items........................................................................................................................................ 23
Answer alternatives ..................................................................................................................................... 27
Bipolar versus Unipolar answer alternatives (DIRECR EXAM MATERIAL) .................................................... 29
Well designed Likert Items: Symmetric and Balanced ................................................................................ 29
Questionnaire Construction: Layout ........................................................................................................... 30
Introductory part ......................................................................................................................................... 30
Instructions .................................................................................................................................................. 30
Order of the questions ................................................................................................................................ 31
Layout .......................................................................................................................................................... 31
Concluding part ........................................................................................................................................... 31
Pilot study/ pre-testing ................................................................................................................................ 32
Exam questions............................................................................................................................................ 32
In between: Covariance and Correlations ....................................................................................................... 34
Analysis of questionnaire data .................................................................................................................... 34
Measurement Levels ................................................................................................................................... 34
Covariance ................................................................................................................................................... 36
Correlation: standardized covariance! ........................................................................................................ 39
Correlation: formula .................................................................................................................................... 40
Exam questions............................................................................................................................................ 42
How does the standardization work? .......................................................................................................... 43
Lecture 4: Properties of a Linear Correlation; Prominent Dimensionality Reduction Techniques ................... 45
, Linear Transformations ............................................................................................................................... 45
Linear combinations .................................................................................................................................... 47
Exam questions............................................................................................................................................ 51
Some practical advice .................................................................................................................................. 52
Prominent Dimensionality Reduction Techniques ...................................................................................... 52
FA versus PCA .............................................................................................................................................. 53
Principal Component Analysis (PCA) ........................................................................................................... 53
Lecture 5: Principal component analysis Part 1 .............................................................................................. 54
Testing the assumptions .............................................................................................................................. 54
Principal Component Analysis ..................................................................................................................... 57
Variance partitioning in PCA ........................................................................................................................ 58
Component scores and component loadings .............................................................................................. 58
Steps in PCA (1)............................................................................................................................................ 58
Important for exam!! ................................................................................................................................... 60
Methods for selecting the number of dimensions (PCA,EFA) ..................................................................... 61
Theory .......................................................................................................................................................... 62
Examples ...................................................................................................................................................... 63
What is component loading and eigenvalues? ............................................................................................ 63
Dividing the questionnaire into (sub)scales ................................................................................................ 65
Exam questions............................................................................................................................................ 65
Lecture 6: Principal Component Analysis (PCA part 2) .................................................................................... 67
The ideal hypothetical case ......................................................................................................................... 68
A nice simple structure – check the rotated solution .................................................................................. 69
Data example ............................................................................................................................................... 70
One-component PCA: Equation ................................................................................................................... 70
PCA with two components .......................................................................................................................... 76
Exam questions............................................................................................................................................ 80
Lecture 7: Rotations, Principal Axis Factoring (PAF)........................................................................................ 82
Rotation ....................................................................................................................................................... 84
Summary and manual for a PCA .................................................................................................................. 88
Factor Analysis introduction ........................................................................................................................ 89
Exam question ............................................................................................................................................. 95
Lecture 8: Comparison PCA versus EFA, Concept of CFA ................................................................................. 96
What are dimensionality reduction techniques about? .............................................................................. 96
PCA versus PAF ............................................................................................................................................ 97
Determine validity: analytical steps: ........................................................................................................... 99
Prominent Dimensionality Reduction Techniques .................................................................................... 100
Exploratory versus Confirmatory FA (EFA versus CFA) .............................................................................. 100
Exam Questions ......................................................................................................................................... 101
Lecture 9: Construct and Criterion-Oriented Validity; MTMM matrix ........................................................... 104
Method 1: (deductive) Construct method ................................................................................................. 104
Construct validity: nomological network ................................................................................................... 104
Construct Validity (coverage of the content) ............................................................................................ 105
Construct Method: .................................................................................................................................... 105
Evaluation of Construct Validity: MTMM Matrix ....................................................................................... 106
Multitrait-Multimethod Matrix (MTMM) .................................................................................................. 107
Method 6: (inductive): external method (criterion oriented) ................................................................... 109
Attenuation of Correlation (verzwakking is vertaling van attenuation) .................................................... 111
Exam Question........................................................................................................................................... 112
, Lecture 10: Measurement Error: CTT versus MTT; Forms or Reliability; Cronbach’s A .................................. 114
Error type 1: Random Measurement Error ............................................................................................... 115
Error type 2: Systematic Measurement Error ........................................................................................... 115
How does measurement error affect correlation? .................................................................................... 117
Classical Test Theory (CTT) Versus Modern Test Theory (MTT) ................................................................ 117
Conceptual Framework of CTT .................................................................................................................. 118
Reliability ................................................................................................................................................... 119
Reliability (Statistical Definition) ............................................................................................................... 119
Heterogeneity among the respondents and reliability ............................................................................. 120
How to estimate measurement error in reliability assessment? .............................................................. 120
Contra-indicative items ............................................................................................................................. 126
Exam Question........................................................................................................................................... 126
Lecture 11: Reliability in SPSS: Spearman-Brown Formula; Correction for attenuation (verzwakking) ......... 128
Evaluation of the reliability of the overall scale ........................................................................................ 128
Determining reliability in SPSS................................................................................................................... 129
Contra-indicative (RW= reverse worded) items ....................................................................................... 129
Evaluation of the reliability of the overall scale ........................................................................................ 130
Reliability and the length of a questionnaire/test ..................................................................................... 132
Remedy: the spearman-brown ‘prophecy’ formula .................................................................................. 133
Exam questions.......................................................................................................................................... 141
Prelude
In our field, our attention is often drawn to human traits, beliefs or behaviors.
However…
• Human traits and beliefs are often nog immediately observable
• To make them measurable, we need measurement instruments
• Frequently used: Questionnaires (e.g. personality tests, wellbeing)
• To ensure the validity and reliability of a questionnaire, a number of quality criteria
must be met
About…
1. Operationalize: converting non-directly observable traits, beliefs or behaviors into
concrete questions
2. Assess dimensionality ‘clustering’ questions into meaningful scales
3. Evaluate validity and reliability: investigating the (technical) quality of the scale
scores
Aantekeningen
Blok 3, 2024
Tilburg University
,Inhoudsopgave
Construction and Analysis of Questionnaires ................................................................................................... 1
Prelude............................................................................................................................................................. 4
Lecture 1: Why Questionnaires? ...................................................................................................................... 5
Definition of a Questionnaire ........................................................................................................................ 5
What is an ‘Item’? .......................................................................................................................................... 6
What is the purpose of a questionnaire? ...................................................................................................... 6
Measurement in technical sciences ............................................................................................................ 10
Measurement in social sciences .................................................................................................................. 10
Validity ......................................................................................................................................................... 10
Reliability ..................................................................................................................................................... 10
Content of questionnaires ........................................................................................................................... 11
Modes and question types .......................................................................................................................... 11
Former exam questions ............................................................................................................................... 11
Lecture 2: Making constructs measurable ...................................................................................................... 12
What are constructs? .................................................................................................................................. 12
Making constructs measurable ................................................................................................................... 13
Six established methods of Scale Construction ........................................................................................... 14
1. Construct Method .............................................................................................................................. 15
2. Facet Design Method ......................................................................................................................... 17
3. Rational Method ................................................................................................................................ 18
4. Prototypical Method .......................................................................................................................... 19
5. Internal Method ................................................................................................................................. 19
6. External Method ................................................................................................................................ 20
Exam questions............................................................................................................................................ 21
Lecture 3: Questionnaires in Practice ............................................................................................................. 23
Formulating items........................................................................................................................................ 23
Answer alternatives ..................................................................................................................................... 27
Bipolar versus Unipolar answer alternatives (DIRECR EXAM MATERIAL) .................................................... 29
Well designed Likert Items: Symmetric and Balanced ................................................................................ 29
Questionnaire Construction: Layout ........................................................................................................... 30
Introductory part ......................................................................................................................................... 30
Instructions .................................................................................................................................................. 30
Order of the questions ................................................................................................................................ 31
Layout .......................................................................................................................................................... 31
Concluding part ........................................................................................................................................... 31
Pilot study/ pre-testing ................................................................................................................................ 32
Exam questions............................................................................................................................................ 32
In between: Covariance and Correlations ....................................................................................................... 34
Analysis of questionnaire data .................................................................................................................... 34
Measurement Levels ................................................................................................................................... 34
Covariance ................................................................................................................................................... 36
Correlation: standardized covariance! ........................................................................................................ 39
Correlation: formula .................................................................................................................................... 40
Exam questions............................................................................................................................................ 42
How does the standardization work? .......................................................................................................... 43
Lecture 4: Properties of a Linear Correlation; Prominent Dimensionality Reduction Techniques ................... 45
, Linear Transformations ............................................................................................................................... 45
Linear combinations .................................................................................................................................... 47
Exam questions............................................................................................................................................ 51
Some practical advice .................................................................................................................................. 52
Prominent Dimensionality Reduction Techniques ...................................................................................... 52
FA versus PCA .............................................................................................................................................. 53
Principal Component Analysis (PCA) ........................................................................................................... 53
Lecture 5: Principal component analysis Part 1 .............................................................................................. 54
Testing the assumptions .............................................................................................................................. 54
Principal Component Analysis ..................................................................................................................... 57
Variance partitioning in PCA ........................................................................................................................ 58
Component scores and component loadings .............................................................................................. 58
Steps in PCA (1)............................................................................................................................................ 58
Important for exam!! ................................................................................................................................... 60
Methods for selecting the number of dimensions (PCA,EFA) ..................................................................... 61
Theory .......................................................................................................................................................... 62
Examples ...................................................................................................................................................... 63
What is component loading and eigenvalues? ............................................................................................ 63
Dividing the questionnaire into (sub)scales ................................................................................................ 65
Exam questions............................................................................................................................................ 65
Lecture 6: Principal Component Analysis (PCA part 2) .................................................................................... 67
The ideal hypothetical case ......................................................................................................................... 68
A nice simple structure – check the rotated solution .................................................................................. 69
Data example ............................................................................................................................................... 70
One-component PCA: Equation ................................................................................................................... 70
PCA with two components .......................................................................................................................... 76
Exam questions............................................................................................................................................ 80
Lecture 7: Rotations, Principal Axis Factoring (PAF)........................................................................................ 82
Rotation ....................................................................................................................................................... 84
Summary and manual for a PCA .................................................................................................................. 88
Factor Analysis introduction ........................................................................................................................ 89
Exam question ............................................................................................................................................. 95
Lecture 8: Comparison PCA versus EFA, Concept of CFA ................................................................................. 96
What are dimensionality reduction techniques about? .............................................................................. 96
PCA versus PAF ............................................................................................................................................ 97
Determine validity: analytical steps: ........................................................................................................... 99
Prominent Dimensionality Reduction Techniques .................................................................................... 100
Exploratory versus Confirmatory FA (EFA versus CFA) .............................................................................. 100
Exam Questions ......................................................................................................................................... 101
Lecture 9: Construct and Criterion-Oriented Validity; MTMM matrix ........................................................... 104
Method 1: (deductive) Construct method ................................................................................................. 104
Construct validity: nomological network ................................................................................................... 104
Construct Validity (coverage of the content) ............................................................................................ 105
Construct Method: .................................................................................................................................... 105
Evaluation of Construct Validity: MTMM Matrix ....................................................................................... 106
Multitrait-Multimethod Matrix (MTMM) .................................................................................................. 107
Method 6: (inductive): external method (criterion oriented) ................................................................... 109
Attenuation of Correlation (verzwakking is vertaling van attenuation) .................................................... 111
Exam Question........................................................................................................................................... 112
, Lecture 10: Measurement Error: CTT versus MTT; Forms or Reliability; Cronbach’s A .................................. 114
Error type 1: Random Measurement Error ............................................................................................... 115
Error type 2: Systematic Measurement Error ........................................................................................... 115
How does measurement error affect correlation? .................................................................................... 117
Classical Test Theory (CTT) Versus Modern Test Theory (MTT) ................................................................ 117
Conceptual Framework of CTT .................................................................................................................. 118
Reliability ................................................................................................................................................... 119
Reliability (Statistical Definition) ............................................................................................................... 119
Heterogeneity among the respondents and reliability ............................................................................. 120
How to estimate measurement error in reliability assessment? .............................................................. 120
Contra-indicative items ............................................................................................................................. 126
Exam Question........................................................................................................................................... 126
Lecture 11: Reliability in SPSS: Spearman-Brown Formula; Correction for attenuation (verzwakking) ......... 128
Evaluation of the reliability of the overall scale ........................................................................................ 128
Determining reliability in SPSS................................................................................................................... 129
Contra-indicative (RW= reverse worded) items ....................................................................................... 129
Evaluation of the reliability of the overall scale ........................................................................................ 130
Reliability and the length of a questionnaire/test ..................................................................................... 132
Remedy: the spearman-brown ‘prophecy’ formula .................................................................................. 133
Exam questions.......................................................................................................................................... 141
Prelude
In our field, our attention is often drawn to human traits, beliefs or behaviors.
However…
• Human traits and beliefs are often nog immediately observable
• To make them measurable, we need measurement instruments
• Frequently used: Questionnaires (e.g. personality tests, wellbeing)
• To ensure the validity and reliability of a questionnaire, a number of quality criteria
must be met
About…
1. Operationalize: converting non-directly observable traits, beliefs or behaviors into
concrete questions
2. Assess dimensionality ‘clustering’ questions into meaningful scales
3. Evaluate validity and reliability: investigating the (technical) quality of the scale
scores
