Survey Summary (Lectures + additional reading materials)
Lecture 1 – introduction ........................................................................................... 2
Literature week 1.................................................................................................. 5
Lecture 2 – introduction to Surveys ........................................................................... 9
Literature week 2.................................................................................................18
Lecture 3 - Pretesting surveys .................................................................................22
Literature week 3.................................................................................................25
Lecture 4 – Pretesting..............................................................................................29
Literature week 4.................................................................................................33
Lecture 5 & 6 – Factor Analysis + Multi-solution factor analysis ..................................39
Factor analysis....................................................................................................40
Multi-Solution factor analysis...............................................................................45
Overview for the exam.............................................................................................48
, Lecture 1 – introduction
Steps on how to design a survey
1. What do you want to measure
2. From theory to questions and answers
3. Phrase specific items
4. Develop survey
5. Pretest survey
6. Run survey
Step 1: what do you want to measure?
- Check the exact research objectives
o E.g. with supervisors
o Specification of your ‘rough’ assignment
Manifest and latent variables
- Manifest: can be directly observed
o E.g., height, hair and color
- Latent variables: can only be observed indirectly
o E.g., wealth, intelligence, attitudes as political efficacy, comprehensibility, risk
perception, etc.
o Multiple questions to measure, because:
▪ The concepts are abstract and multi-faceted
▪ When you ask multiple questions about the same construct you will be
able to establish that you have measured one underlying thing
▪ You can detect/decrease the influence of unsystematic errors (people
providing the wrong answer)
o Self-report measures of a latent construct (for example, depression)
▪ Measurements that represent a set of indicators of the latent
construct
▪ If you score high on Y, then this should reflect in A, B, C, etc.
Uni vs multidimensional scales
2
,Unidimensional scale: used to describe a specific type of measurement scale. A
unidimensional measurement scale has only one (“uni”) dimension. It can be represented by
a single number line.
- E.g., height of people, weight of cars, IQ, volume of liquid
- It can also refer to measuring a single ability, attribute, construct or skill.
o E.g., a unidimensional mathematical test would be designed to measure only
mathematical ability (and not, other non-mathematical subject or concepts)
Multidimensional scale: a visual representation of distances or similarities between the set
of objects. Objects that are more similar are closer together on a graph than objects that are
less similar.
- The “multidimensional” is partly because you aren’t limited to two dimensional
graphs or data.
Example of a scale
- Is the following scale an okay scale to measure how much people like to eat
vegetables?
o Please indicate to what extent you agree with the following items (1 =
completely disagree, 7 = completely agree)
▪ I love eating bananas
▪ I really like strawberries
▪ I find apples delicious things to eat
▪ I enjoy eating a good orange
- → The scale does not measure what we want to measure (not valid)
- → Though, the scale’s reliability may still be okay
-
Step 2: from theory to questions and answers
- Option 1: existing scales
o Need for cognition
o Privacy concerns
- Advantage:
o Validated
- Disadvantages
o Language/translations; could be weirdly phrased
o Not all scientists are survey methodologists
What if there is no scale available? → develop your own scale
- Internal method (inductive): Many items are used and through statistical grouping
techniques it is decided after the fact which ones were relevant
3
, - Facet method (deductive): Instrument should fully represent each dimension of the
construct that is intended to be measured
o Teacher quality → capacity to explain (examples, good pace), approachability
(open to questions, responds quickly to emails)
Step 3: phrase specific items
Conventional wisdom (Krosnick & Presser, 2010)
1. Use simple, familiar words (avoid technical terms, jargon and slang)
2. Use simple syntax
3. Avoid words with ambiguous meanings; aim for wording that all respondents will
interpret in the same way
4. Strive for wording that is specific and concrete
o (As opposed to general and abstract)
5. Make response options exhaustive and mutually exclusive
6. Avoid leading or loaded questions that push respondents toward an answer
7. Ask about one thing at a time (avoid double- questions)
8. Avoid questions with single or double negations
Step 4: develop the survey & choose the lay out
- Conventional wisdom contains advice about how to optimize question order
(Krosnick & Presser, 2010)
1. Early questions should be easy and pleasant to answer and should build rapport
between the respondent and the researcher
2. Questions at the very beginning of a questionnaire should be explicitly address
the topic of the survey, as it was described to the respondent prior to the
interview
3. Questions on the same topic should be grouped together
4. Questions on the same topic should proceed from general to specific
5. Questions on sensitive topics that might make respondents uncomfortable should
be placed at the end of the questionnaire
6. Filter questions should be included, to avoid asking respondents questions that
do not apply to them
Step 5: pretest the survey
- Cognitive interview
- Informal debriefing
- → lecture 3
Step 6: run the survey
- May also be step 7, 8 or 9 depending on the results of step 5
4
Lecture 1 – introduction ........................................................................................... 2
Literature week 1.................................................................................................. 5
Lecture 2 – introduction to Surveys ........................................................................... 9
Literature week 2.................................................................................................18
Lecture 3 - Pretesting surveys .................................................................................22
Literature week 3.................................................................................................25
Lecture 4 – Pretesting..............................................................................................29
Literature week 4.................................................................................................33
Lecture 5 & 6 – Factor Analysis + Multi-solution factor analysis ..................................39
Factor analysis....................................................................................................40
Multi-Solution factor analysis...............................................................................45
Overview for the exam.............................................................................................48
, Lecture 1 – introduction
Steps on how to design a survey
1. What do you want to measure
2. From theory to questions and answers
3. Phrase specific items
4. Develop survey
5. Pretest survey
6. Run survey
Step 1: what do you want to measure?
- Check the exact research objectives
o E.g. with supervisors
o Specification of your ‘rough’ assignment
Manifest and latent variables
- Manifest: can be directly observed
o E.g., height, hair and color
- Latent variables: can only be observed indirectly
o E.g., wealth, intelligence, attitudes as political efficacy, comprehensibility, risk
perception, etc.
o Multiple questions to measure, because:
▪ The concepts are abstract and multi-faceted
▪ When you ask multiple questions about the same construct you will be
able to establish that you have measured one underlying thing
▪ You can detect/decrease the influence of unsystematic errors (people
providing the wrong answer)
o Self-report measures of a latent construct (for example, depression)
▪ Measurements that represent a set of indicators of the latent
construct
▪ If you score high on Y, then this should reflect in A, B, C, etc.
Uni vs multidimensional scales
2
,Unidimensional scale: used to describe a specific type of measurement scale. A
unidimensional measurement scale has only one (“uni”) dimension. It can be represented by
a single number line.
- E.g., height of people, weight of cars, IQ, volume of liquid
- It can also refer to measuring a single ability, attribute, construct or skill.
o E.g., a unidimensional mathematical test would be designed to measure only
mathematical ability (and not, other non-mathematical subject or concepts)
Multidimensional scale: a visual representation of distances or similarities between the set
of objects. Objects that are more similar are closer together on a graph than objects that are
less similar.
- The “multidimensional” is partly because you aren’t limited to two dimensional
graphs or data.
Example of a scale
- Is the following scale an okay scale to measure how much people like to eat
vegetables?
o Please indicate to what extent you agree with the following items (1 =
completely disagree, 7 = completely agree)
▪ I love eating bananas
▪ I really like strawberries
▪ I find apples delicious things to eat
▪ I enjoy eating a good orange
- → The scale does not measure what we want to measure (not valid)
- → Though, the scale’s reliability may still be okay
-
Step 2: from theory to questions and answers
- Option 1: existing scales
o Need for cognition
o Privacy concerns
- Advantage:
o Validated
- Disadvantages
o Language/translations; could be weirdly phrased
o Not all scientists are survey methodologists
What if there is no scale available? → develop your own scale
- Internal method (inductive): Many items are used and through statistical grouping
techniques it is decided after the fact which ones were relevant
3
, - Facet method (deductive): Instrument should fully represent each dimension of the
construct that is intended to be measured
o Teacher quality → capacity to explain (examples, good pace), approachability
(open to questions, responds quickly to emails)
Step 3: phrase specific items
Conventional wisdom (Krosnick & Presser, 2010)
1. Use simple, familiar words (avoid technical terms, jargon and slang)
2. Use simple syntax
3. Avoid words with ambiguous meanings; aim for wording that all respondents will
interpret in the same way
4. Strive for wording that is specific and concrete
o (As opposed to general and abstract)
5. Make response options exhaustive and mutually exclusive
6. Avoid leading or loaded questions that push respondents toward an answer
7. Ask about one thing at a time (avoid double- questions)
8. Avoid questions with single or double negations
Step 4: develop the survey & choose the lay out
- Conventional wisdom contains advice about how to optimize question order
(Krosnick & Presser, 2010)
1. Early questions should be easy and pleasant to answer and should build rapport
between the respondent and the researcher
2. Questions at the very beginning of a questionnaire should be explicitly address
the topic of the survey, as it was described to the respondent prior to the
interview
3. Questions on the same topic should be grouped together
4. Questions on the same topic should proceed from general to specific
5. Questions on sensitive topics that might make respondents uncomfortable should
be placed at the end of the questionnaire
6. Filter questions should be included, to avoid asking respondents questions that
do not apply to them
Step 5: pretest the survey
- Cognitive interview
- Informal debriefing
- → lecture 3
Step 6: run the survey
- May also be step 7, 8 or 9 depending on the results of step 5
4
