METHODS week 6
Survey
A survey is a series of formatted questions delivered to a defined sample of people with
the expectation of responses immediately or within a few days. With appropriate sampling,
we can generalize with a known statistical confidence from a sample to a wider
population
+ Relatively low cost
+ Relatively fast delivery
+ Rapid data processing
+ Can reach large populations
+ Multiple methods
- Generally limited to scaled or check-list questions (you can find out what is going on but
not why)
- No control over response rate (how many people respond)
- Provides information more than understanding
- Increasing public resistance (survey fatigue = people are getting sick of surveys)
- Difficult to explore issues in depth
Cross-sectional
Ask 1 group of people (respondents) the same questions at 1 point in time
Longitudinal (you can say something about how opinions develop over time)
Observations are taken more than once
You examine differences/relationships over time
Not able to detect causal relationships
Takes into account influence of time
Types of longitudinal surveys: trend/cohort/panel/cross-lagged
Trend
Longitudinal study examines changes in population across time
Each study collects data from different individuals
Not clear why change occurred
Cohort
Longitudinal study examines changes in a cohort across time
Each study collects data from different individuals within the same
cohort
Not clear why change occurred but provides more detail than trend
Panel
Longitudinal study examines changes in individuals across time
Each study collects data from the same individuals
More in-depth analysis (expensive)
Still can’t provide causality and effect but more detail than cohort
To determine causality you need a true experiment. Because in an experiment only the
stimulus changes. You have control over all the other conditions, you make sure they are
the same. You are sure that the stimulus causes the change. Panel and cross-lagged
surveys may give some indication of causality. The internal validity of a survey is low, if
we have a large and random sample the external validity can be really high.
- A cross-sectional study is the least expensive to implement, but cannot determine cause
and effect.
1
, - Longitudinal studies are more expensive to implement, but provide a better sense of
cause and effect.
Survey mode
With interviewer Response Speed Costs Complexity ques-
tionairre
Face-to-face Good Slow (Very) high (Very) high
Phone Reasonable Fast High (Very) high
Without Inter-
viewer
Paper-and-pencil Low Slow Low/moderate Low
Email/web Low Very fast Very low Very high
Types of questions
- Dichotomous: yes/no
- Open ended
- Multiple choice (mutually exclusive)
- Multiple choice (check only)
- Rank order
- Likert scale
- Semantic differential scale (two opposite words)
Problems with survey wording
Leading questions
It provides an indication of what is the right answer
Double negative wording
Overlap answer options
Double-barreled question
There are multiple questions and only one answer scale
Social desirability bias: The respondent thinks that the interviewer wants to hear a
certain thing and the respondent doesn’t want to go against that social norm.
Prevent social desirability: Indirect questioning (third party reference) and show that it
is okay to answer in a way that is not socially desirable.
Pre-testing
- Survey wording
- Aesthetics and design of questionnaire
- Logic and flow of wording
- Words - and their specific meanings
- Length/time it takes to complete questionnaire
How to pre-testing
- Ask respondents to fill in questionnaire and add evaluation questions
- Cognitive interviewing: use think-aloud method and ask respondents what they think
when filling in questionnaire
- Ask your supervisor, fellow students or colleague for feedback
2
Survey
A survey is a series of formatted questions delivered to a defined sample of people with
the expectation of responses immediately or within a few days. With appropriate sampling,
we can generalize with a known statistical confidence from a sample to a wider
population
+ Relatively low cost
+ Relatively fast delivery
+ Rapid data processing
+ Can reach large populations
+ Multiple methods
- Generally limited to scaled or check-list questions (you can find out what is going on but
not why)
- No control over response rate (how many people respond)
- Provides information more than understanding
- Increasing public resistance (survey fatigue = people are getting sick of surveys)
- Difficult to explore issues in depth
Cross-sectional
Ask 1 group of people (respondents) the same questions at 1 point in time
Longitudinal (you can say something about how opinions develop over time)
Observations are taken more than once
You examine differences/relationships over time
Not able to detect causal relationships
Takes into account influence of time
Types of longitudinal surveys: trend/cohort/panel/cross-lagged
Trend
Longitudinal study examines changes in population across time
Each study collects data from different individuals
Not clear why change occurred
Cohort
Longitudinal study examines changes in a cohort across time
Each study collects data from different individuals within the same
cohort
Not clear why change occurred but provides more detail than trend
Panel
Longitudinal study examines changes in individuals across time
Each study collects data from the same individuals
More in-depth analysis (expensive)
Still can’t provide causality and effect but more detail than cohort
To determine causality you need a true experiment. Because in an experiment only the
stimulus changes. You have control over all the other conditions, you make sure they are
the same. You are sure that the stimulus causes the change. Panel and cross-lagged
surveys may give some indication of causality. The internal validity of a survey is low, if
we have a large and random sample the external validity can be really high.
- A cross-sectional study is the least expensive to implement, but cannot determine cause
and effect.
1
, - Longitudinal studies are more expensive to implement, but provide a better sense of
cause and effect.
Survey mode
With interviewer Response Speed Costs Complexity ques-
tionairre
Face-to-face Good Slow (Very) high (Very) high
Phone Reasonable Fast High (Very) high
Without Inter-
viewer
Paper-and-pencil Low Slow Low/moderate Low
Email/web Low Very fast Very low Very high
Types of questions
- Dichotomous: yes/no
- Open ended
- Multiple choice (mutually exclusive)
- Multiple choice (check only)
- Rank order
- Likert scale
- Semantic differential scale (two opposite words)
Problems with survey wording
Leading questions
It provides an indication of what is the right answer
Double negative wording
Overlap answer options
Double-barreled question
There are multiple questions and only one answer scale
Social desirability bias: The respondent thinks that the interviewer wants to hear a
certain thing and the respondent doesn’t want to go against that social norm.
Prevent social desirability: Indirect questioning (third party reference) and show that it
is okay to answer in a way that is not socially desirable.
Pre-testing
- Survey wording
- Aesthetics and design of questionnaire
- Logic and flow of wording
- Words - and their specific meanings
- Length/time it takes to complete questionnaire
How to pre-testing
- Ask respondents to fill in questionnaire and add evaluation questions
- Cognitive interviewing: use think-aloud method and ask respondents what they think
when filling in questionnaire
- Ask your supervisor, fellow students or colleague for feedback
2
