Week 1........................................................................................................2
Lecture. Introduction to surveys...............................................................2
Blog on of biased survey questions (Toepoel, 2016)................................4
Video on Latent variables.........................................................................5
Week 2........................................................................................................6
Lecture. The psychology of answering survey questions.........................6
Blog on Survey introductions..................................................................13
International handbook on survey methodology, Chapter 8 (Fowler &
Cosenza, 2012).......................................................................................13
Developing the survey – Questions and answers (Toepoel, 2016).........15
Week 3......................................................................................................17
Lecture. Pretesting.................................................................................17
The threat of satisficing in Surveys: The shortcuts respondents take in
answering questions (Krosnick, 2000)....................................................18
Questionnaire Pretesting Methods (Ikart, 2018).....................................20
WEEK 4......................................................................................................23
Lecture. Online surveys..........................................................................23
Programming the survey (Toepoel, 2016)..............................................25
WEEK 5......................................................................................................29
Lecture. Factor Analysis 1......................................................................29
Practice Factor Analysis..........................................................................40
WEEK 6......................................................................................................49
Lecture. Factor analysis 2.......................................................................49
Lecture 7. Review session.........................................................................53
Quiz...........................................................................................................55
Answers quiz.............................................................................................56
,Week 1.
Lecture. Introduction to surveys
How to design a survey?
Step 1. What do you want to measure?
Check the exact research objectives. Specification of your rough assignment.
Manifest variables: can be directly observed, for example height, hair color.
Latent variables: can only be observed indirectly, for example wealth, intelligence, attitudes as
political efficacy, being introvert, comprehensibility, risk perception, etc.
Those variables need multiple questions, because the concepts are abstract and multi-faceted.
When you ask multiple questions about the same construct you will at very least be able to establish
that you have measured one underlying thing. You can detect/decrease the influence of
unsystematic errors (people providing the wrong answer).
Self-report measures of a latent construct (for example depression). Measurements that represent a
set of indicators of the latent construct. If you score high on U, then this should reflect in A, B, C, etc.
Unidimensional scales versus multidimensional scales:
For example, is the following scale an okay scale to measure how much people like to eat
vegetables?
,Please indicate to what extent you agree with the following items (1= completely disagree, 7=
completely agree).
- I love eating bananas
- I really like strawberries
- I find apples delicious things to eat
- I enjoy eating a good orange
This scale does not measure what we want to measure (it measures fruit instead of vegetables) = not
valid.
Though, the scale's reliability may still be ok (consistent answers).
Step 2. From theory to questions and answers
There are two options to choose a scale.
Option 1: using existing scales, like the need for cognition scale or privacy concern scale.
- Advantage: the scales are validated.
- Disadvantages: language or translations issues (may lead to different/vague questions), not
all scientists are survey methodologists (some questions are difficult), or there may be no
scale available.
Option 2: developing your own scale or items.
Internal method (inductive): this means that many items are used and through statistical grouping
techniques it is decided which items are relevant.
Facet method (deductive): instrument should fully represent each dimension of the construct that is
intended to be measured. For example, what could be facets of teacher quality?
- Capacity to explain (=construct): giving examples, uses a good pace while talking, etc.
(=facets).
- Approachability (=construct): open to questions, responds to emails quickly, etc. (=facets).
- Etc.
Step 3. Phrase specific items
Conventional wisdom: how to phrase and order questions (Krosnick & Presser, 2010).
How to phrase questions:
- Use simple, familiar words (avoid technical terms, jargon, and slang).
- Use simply syntax.
- Avoid words with ambiguous meanings: aim for wording that all respondents will interpret in
the same way.
- Strive for wording that is specific and concrete (as opposed to general and abstract).
- Make response options exhaustive and mutually exclusive.
, - Avoid leading or loaded questions that push respondents toward an answer.
- Ask about one thing at a time (avoid double-barreled questions).
- Avoid questions with single or double negations.
Step 4. Develop the survey
Conventional wisdom: how to optimize question order (Krosnick & Presser, 2010):
- Early questions should be easy and pleasant to answer and should build rapport between
the respondent and the researcher.
- Questions at the very beginning of a questionnaire should explicitly address the topic of the
survey, as it was described to the respondent prior to the interview.
- Questions on the same topic should be grouped together.
- Questions on the same topic should proceed from general to specific.
- Re.
- Filter questions should be included, to avoid asking respondents questions that do not apply
to them.
Step 5. Pretest the survey
Cognitive interview, informal debriefing, etc. (lecture 3).
Step 6. Run
Take home messages:
- Specify what you aim to measure.
- Constructs that involve much abstraction (e.g. attitudes) require multiple items.
- Check if there are existing scales that you can use, but always be critical.
- If you phrase your own questions: think about the constructs and their facets first.
Blog on of biased survey questions (Toepoel, 2016)
Biased survey: a survey that encompasses errors caused by the design of the survey and its
questions. It's important for you, the survey creator, to create survey questions that don't change
the survey’s outcome. Things to consider:
- The way questions are worded.
- The structure of the survey.
- The design, style and colors of the survey.
A biased survey can lead to survey response bias and higher than normal drop-out rates.
Lecture. Introduction to surveys...............................................................2
Blog on of biased survey questions (Toepoel, 2016)................................4
Video on Latent variables.........................................................................5
Week 2........................................................................................................6
Lecture. The psychology of answering survey questions.........................6
Blog on Survey introductions..................................................................13
International handbook on survey methodology, Chapter 8 (Fowler &
Cosenza, 2012).......................................................................................13
Developing the survey – Questions and answers (Toepoel, 2016).........15
Week 3......................................................................................................17
Lecture. Pretesting.................................................................................17
The threat of satisficing in Surveys: The shortcuts respondents take in
answering questions (Krosnick, 2000)....................................................18
Questionnaire Pretesting Methods (Ikart, 2018).....................................20
WEEK 4......................................................................................................23
Lecture. Online surveys..........................................................................23
Programming the survey (Toepoel, 2016)..............................................25
WEEK 5......................................................................................................29
Lecture. Factor Analysis 1......................................................................29
Practice Factor Analysis..........................................................................40
WEEK 6......................................................................................................49
Lecture. Factor analysis 2.......................................................................49
Lecture 7. Review session.........................................................................53
Quiz...........................................................................................................55
Answers quiz.............................................................................................56
,Week 1.
Lecture. Introduction to surveys
How to design a survey?
Step 1. What do you want to measure?
Check the exact research objectives. Specification of your rough assignment.
Manifest variables: can be directly observed, for example height, hair color.
Latent variables: can only be observed indirectly, for example wealth, intelligence, attitudes as
political efficacy, being introvert, comprehensibility, risk perception, etc.
Those variables need multiple questions, because the concepts are abstract and multi-faceted.
When you ask multiple questions about the same construct you will at very least be able to establish
that you have measured one underlying thing. You can detect/decrease the influence of
unsystematic errors (people providing the wrong answer).
Self-report measures of a latent construct (for example depression). Measurements that represent a
set of indicators of the latent construct. If you score high on U, then this should reflect in A, B, C, etc.
Unidimensional scales versus multidimensional scales:
For example, is the following scale an okay scale to measure how much people like to eat
vegetables?
,Please indicate to what extent you agree with the following items (1= completely disagree, 7=
completely agree).
- I love eating bananas
- I really like strawberries
- I find apples delicious things to eat
- I enjoy eating a good orange
This scale does not measure what we want to measure (it measures fruit instead of vegetables) = not
valid.
Though, the scale's reliability may still be ok (consistent answers).
Step 2. From theory to questions and answers
There are two options to choose a scale.
Option 1: using existing scales, like the need for cognition scale or privacy concern scale.
- Advantage: the scales are validated.
- Disadvantages: language or translations issues (may lead to different/vague questions), not
all scientists are survey methodologists (some questions are difficult), or there may be no
scale available.
Option 2: developing your own scale or items.
Internal method (inductive): this means that many items are used and through statistical grouping
techniques it is decided which items are relevant.
Facet method (deductive): instrument should fully represent each dimension of the construct that is
intended to be measured. For example, what could be facets of teacher quality?
- Capacity to explain (=construct): giving examples, uses a good pace while talking, etc.
(=facets).
- Approachability (=construct): open to questions, responds to emails quickly, etc. (=facets).
- Etc.
Step 3. Phrase specific items
Conventional wisdom: how to phrase and order questions (Krosnick & Presser, 2010).
How to phrase questions:
- Use simple, familiar words (avoid technical terms, jargon, and slang).
- Use simply syntax.
- Avoid words with ambiguous meanings: aim for wording that all respondents will interpret in
the same way.
- Strive for wording that is specific and concrete (as opposed to general and abstract).
- Make response options exhaustive and mutually exclusive.
, - Avoid leading or loaded questions that push respondents toward an answer.
- Ask about one thing at a time (avoid double-barreled questions).
- Avoid questions with single or double negations.
Step 4. Develop the survey
Conventional wisdom: how to optimize question order (Krosnick & Presser, 2010):
- Early questions should be easy and pleasant to answer and should build rapport between
the respondent and the researcher.
- Questions at the very beginning of a questionnaire should explicitly address the topic of the
survey, as it was described to the respondent prior to the interview.
- Questions on the same topic should be grouped together.
- Questions on the same topic should proceed from general to specific.
- Re.
- Filter questions should be included, to avoid asking respondents questions that do not apply
to them.
Step 5. Pretest the survey
Cognitive interview, informal debriefing, etc. (lecture 3).
Step 6. Run
Take home messages:
- Specify what you aim to measure.
- Constructs that involve much abstraction (e.g. attitudes) require multiple items.
- Check if there are existing scales that you can use, but always be critical.
- If you phrase your own questions: think about the constructs and their facets first.
Blog on of biased survey questions (Toepoel, 2016)
Biased survey: a survey that encompasses errors caused by the design of the survey and its
questions. It's important for you, the survey creator, to create survey questions that don't change
the survey’s outcome. Things to consider:
- The way questions are worded.
- The structure of the survey.
- The design, style and colors of the survey.
A biased survey can lead to survey response bias and higher than normal drop-out rates.
