Summary Design & Analysis questionnaire
Tilburg University 2022
880602-M-6
This document contains all relevant material for the exam, including the lectures, and the mandatory
literature for the Master’s research skill course Design & Analysis Questionnaire.
Celeste Graumans
,Inhoud
Lecture 1; Introduction to the course and to survey research..................................................3
Lecture 1 – Introduction to surveys .....................................................................................3
Lecture 1 – Literature ..........................................................................................................6
Krosnick & Presser, 2010 – Chapter 9 Question & Questionnaire Design page 263 &
264 ..................................................................................................................................6
Blog – 10 Examples of Biased Survey Questions ............................................................7
YouTube video, 2014 – Latent & Manifest Variables .......................................................8
Lecture 1 – Quiz questions .................................................................................................9
Lecture 2; The psychology of answering survey questions ...................................................10
Lecture 2 – The psychology of answering survey questions ..............................................10
Lecture 2 – Literature ........................................................................................................19
Fowler & Cosenza, 2012 – Chapter 8 Writing Effective Questions ................................19
Toepoel, 2016 – Developing the survey: questions and answers ..................................26
Blog – 5 Key messages in an online survey introduction ...............................................31
Lecture 2 – Quiz questions ...............................................................................................31
Lecture 3; Pretesting surveys ...............................................................................................32
Lecture 3 – Pretesting surveys ..........................................................................................32
Lecture 3 – Literature ........................................................................................................34
Krosnick, 2000 – The treat of satisficing in surveys: the shortcuts respondents take in
answering questions......................................................................................................34
Ikart, 2018 page 119 – 133 – Questionnaire pretesting methods: A comparison of
cognitive interviewing and respondent debriefing Vis-à-Vis the study of the adoption of
decision support systems by knowledge workers ..........................................................37
Lecture 4; Online surveys .....................................................................................................40
Lecture 4 – Online surveys ...............................................................................................40
Lecture 4 – Literature ........................................................................................................42
Toepoel, 2016 – Chapter 9 programming the survey – In: doing surveys online ............42
Lecture 5; Factor analysis 1..................................................................................................47
Lecture 5 – Video Intuitive Explanation of Factor Analysis ................................................47
Lecture 5 – Literature ........................................................................................................50
Field, 2013 – Factor Analysis/PCA ................................................................................50
Lecture 6; Factor analysis 2..................................................................................................50
Lecture 6 – Literature ........................................................................................................50
Blog Geert van den Berg, SPSS Tutorials – SPSS Factor Analysis – Beginners Tutorial
......................................................................................................................................50
YouTube video, 2015 – Reliability test: Compute Cronbach’s alpha using SPSS ..........55
Blog – Cronbach’s Alpha (α) using SPSS Statistics .......................................................55
, Lecture 1; Introduction to the course and to survey research
Lecture 1 – Introduction to surveys
How to… design a survey?
Think in concepts.
Steps
1. What do you want to measure?
2. From theory to questions and answers
3. Phrase specific items
4. Develop the survey – order of the questions (lay-out)
5. Pretest the survey – interpretation of the questions
6. Run
Step 1: What do you want to measure?
Check the exact research objectives with your supervisors (Toepoel)
- Specification of your ‘rough’ assignment – focus on a specific part of a subject
- Manifest variables: can be directly observed – e.g., height, hair color
- Latent variables: can only be observed indirectly – e.g., wealth, intelligence,
attitudes as political efficacy, being introverted, comprehensibility, perceived physical
attractiveness
Latent variables
Multiple questions, because:
- The concepts are multi-faceted
- When you ask multiple questions about the same construct you will at the very least
be able to establish that you have measured one underlying thing
- You can detect/decrease the influence of unsystematic errors (people providing the
wrong answer)
Self-report measures of a latent construct (e.g., depression)
- Measurements that represent a set of indicators of the latent construct
- If you score high on Y, then this should reflect in A, B, and C.
- If someone does not experience all indicators, this does not immediately mean that a
person is not depressed.
Example
Is the following scale an okay scale to measure how much people like to eat vegetables?
- Please indicate to what extent you agree with the following items (1 = completely
disagree, 7 = completely agree):
o I love eating bananas
, o I really like strawberries
o I find apples delicious things to eat
o I enjoy eating a good orange
This scale does not measure what we want to measure (not valid) – it is about
vegetables but the statements are about fruit
Step 2: From theory to questions and answers
Option 1: existing scales: the need for cognition, privacy concerns
- Advantage: validated, because other people have used it
- Disadvantages: language/translations, not all scientists are survey methodologists,
and might not fit your data – you have to be critical
Items and their translations (examples)
- I find satisfaction in deliberating hard and for long hours
Iets langdurig en nauwgezet afwegen geeft mij voldoening
There are two questions in one – this is called double-bared questions
- The notion of thinking abstractly is appealing to me
Abstract denken is een bezigheid die mij aanspreekt
This is pretty vague
- I usually end up deliberating about issues even when they do not affect me personally
Gewoonlijk denk ik uitgebreid na over zaken, zelfs wanneer ze mij niet persoonlijk aangaan
What if there is no scale available?
Option 2: developing your own items
- Internal method (inductive): many items are used and through statistical grouping
techniques it is decided after the fact which ones were relevant
- Facet method (deductive): the instrument should fully represent each dimension of
the construct that is intended to be measured
Facet analysis (2; Beehr & Newman, 1978)
The purpose of facet analysis, or more precisely a faceted design, is to delimit and
make explicit the phenomena one wishes to investigate. According to Foa (1968), A facet is
a conceptual dimension underlying a set of variables. It is important in the facet design to
specify all the facets one considers relevant to the domain, regardless of whether all the
properties involved are to be dealt with in the immediate empirical phase of the research.
The meaning of present outcomes is enriched if the outcomes are assessed in the light of
facets conceived as relevant but which remain yet unexplored (Runkel and McGrath, 1972).
The faceted design presented in Table 1 represents a conceptualization of the job
stress-employee health research domain's major facets (dimensions). Examples of elements
that might be included in each facet are included in the table. This facet design is the result
of an inductive-intuitive-inventive process and includes factors the authors think ought to be
studied to understand the phenomena within this research domain.
- Delineating and making explicit the phenomena to be studied
- A conceptual dimension underlying a set of variables
- Specify all facets relevant to the domain, regardless of whether all the properties
involved are addressed in the direct empirical phase of the study.
- A result of inductive-intuitive-inventive process and includes factors that the authors
believe should be studied to understand the phenomena within this research domain.
Tilburg University 2022
880602-M-6
This document contains all relevant material for the exam, including the lectures, and the mandatory
literature for the Master’s research skill course Design & Analysis Questionnaire.
Celeste Graumans
,Inhoud
Lecture 1; Introduction to the course and to survey research..................................................3
Lecture 1 – Introduction to surveys .....................................................................................3
Lecture 1 – Literature ..........................................................................................................6
Krosnick & Presser, 2010 – Chapter 9 Question & Questionnaire Design page 263 &
264 ..................................................................................................................................6
Blog – 10 Examples of Biased Survey Questions ............................................................7
YouTube video, 2014 – Latent & Manifest Variables .......................................................8
Lecture 1 – Quiz questions .................................................................................................9
Lecture 2; The psychology of answering survey questions ...................................................10
Lecture 2 – The psychology of answering survey questions ..............................................10
Lecture 2 – Literature ........................................................................................................19
Fowler & Cosenza, 2012 – Chapter 8 Writing Effective Questions ................................19
Toepoel, 2016 – Developing the survey: questions and answers ..................................26
Blog – 5 Key messages in an online survey introduction ...............................................31
Lecture 2 – Quiz questions ...............................................................................................31
Lecture 3; Pretesting surveys ...............................................................................................32
Lecture 3 – Pretesting surveys ..........................................................................................32
Lecture 3 – Literature ........................................................................................................34
Krosnick, 2000 – The treat of satisficing in surveys: the shortcuts respondents take in
answering questions......................................................................................................34
Ikart, 2018 page 119 – 133 – Questionnaire pretesting methods: A comparison of
cognitive interviewing and respondent debriefing Vis-à-Vis the study of the adoption of
decision support systems by knowledge workers ..........................................................37
Lecture 4; Online surveys .....................................................................................................40
Lecture 4 – Online surveys ...............................................................................................40
Lecture 4 – Literature ........................................................................................................42
Toepoel, 2016 – Chapter 9 programming the survey – In: doing surveys online ............42
Lecture 5; Factor analysis 1..................................................................................................47
Lecture 5 – Video Intuitive Explanation of Factor Analysis ................................................47
Lecture 5 – Literature ........................................................................................................50
Field, 2013 – Factor Analysis/PCA ................................................................................50
Lecture 6; Factor analysis 2..................................................................................................50
Lecture 6 – Literature ........................................................................................................50
Blog Geert van den Berg, SPSS Tutorials – SPSS Factor Analysis – Beginners Tutorial
......................................................................................................................................50
YouTube video, 2015 – Reliability test: Compute Cronbach’s alpha using SPSS ..........55
Blog – Cronbach’s Alpha (α) using SPSS Statistics .......................................................55
, Lecture 1; Introduction to the course and to survey research
Lecture 1 – Introduction to surveys
How to… design a survey?
Think in concepts.
Steps
1. What do you want to measure?
2. From theory to questions and answers
3. Phrase specific items
4. Develop the survey – order of the questions (lay-out)
5. Pretest the survey – interpretation of the questions
6. Run
Step 1: What do you want to measure?
Check the exact research objectives with your supervisors (Toepoel)
- Specification of your ‘rough’ assignment – focus on a specific part of a subject
- Manifest variables: can be directly observed – e.g., height, hair color
- Latent variables: can only be observed indirectly – e.g., wealth, intelligence,
attitudes as political efficacy, being introverted, comprehensibility, perceived physical
attractiveness
Latent variables
Multiple questions, because:
- The concepts are multi-faceted
- When you ask multiple questions about the same construct you will at the very least
be able to establish that you have measured one underlying thing
- You can detect/decrease the influence of unsystematic errors (people providing the
wrong answer)
Self-report measures of a latent construct (e.g., depression)
- Measurements that represent a set of indicators of the latent construct
- If you score high on Y, then this should reflect in A, B, and C.
- If someone does not experience all indicators, this does not immediately mean that a
person is not depressed.
Example
Is the following scale an okay scale to measure how much people like to eat vegetables?
- Please indicate to what extent you agree with the following items (1 = completely
disagree, 7 = completely agree):
o I love eating bananas
, o I really like strawberries
o I find apples delicious things to eat
o I enjoy eating a good orange
This scale does not measure what we want to measure (not valid) – it is about
vegetables but the statements are about fruit
Step 2: From theory to questions and answers
Option 1: existing scales: the need for cognition, privacy concerns
- Advantage: validated, because other people have used it
- Disadvantages: language/translations, not all scientists are survey methodologists,
and might not fit your data – you have to be critical
Items and their translations (examples)
- I find satisfaction in deliberating hard and for long hours
Iets langdurig en nauwgezet afwegen geeft mij voldoening
There are two questions in one – this is called double-bared questions
- The notion of thinking abstractly is appealing to me
Abstract denken is een bezigheid die mij aanspreekt
This is pretty vague
- I usually end up deliberating about issues even when they do not affect me personally
Gewoonlijk denk ik uitgebreid na over zaken, zelfs wanneer ze mij niet persoonlijk aangaan
What if there is no scale available?
Option 2: developing your own items
- Internal method (inductive): many items are used and through statistical grouping
techniques it is decided after the fact which ones were relevant
- Facet method (deductive): the instrument should fully represent each dimension of
the construct that is intended to be measured
Facet analysis (2; Beehr & Newman, 1978)
The purpose of facet analysis, or more precisely a faceted design, is to delimit and
make explicit the phenomena one wishes to investigate. According to Foa (1968), A facet is
a conceptual dimension underlying a set of variables. It is important in the facet design to
specify all the facets one considers relevant to the domain, regardless of whether all the
properties involved are to be dealt with in the immediate empirical phase of the research.
The meaning of present outcomes is enriched if the outcomes are assessed in the light of
facets conceived as relevant but which remain yet unexplored (Runkel and McGrath, 1972).
The faceted design presented in Table 1 represents a conceptualization of the job
stress-employee health research domain's major facets (dimensions). Examples of elements
that might be included in each facet are included in the table. This facet design is the result
of an inductive-intuitive-inventive process and includes factors the authors think ought to be
studied to understand the phenomena within this research domain.
- Delineating and making explicit the phenomena to be studied
- A conceptual dimension underlying a set of variables
- Specify all facets relevant to the domain, regardless of whether all the properties
involved are addressed in the direct empirical phase of the study.
- A result of inductive-intuitive-inventive process and includes factors that the authors
believe should be studied to understand the phenomena within this research domain.
