Survey Workshop Notes
Lecture 1: Valid answers & Proper Questions
Survey Error & Bias
- We want to avoid error and bias.
- Sampling error: random (effects reliability of estimates, not much can be
done), systematic bias error (effects the validity, coverage error in the
sampling frame can be corrected).
- The way you pose/write a question effects the validity of the answers.
Think about where the error stems from to help you work out if it’s random
or systematic.
- Random Error (Sampling error) = how much the sample differs from the
population. We don’t know about the individual level, but overall, we can
expect a good approximation of the opinion.
o Invalid questions lead to mistakes
- Bias (systematic error) = in some systematic way the people responding
to a survey are consistently different from the target population as a
whole. Cannot give a good approximation of opinion but if you have a
systematic bias you’ll know if you're over/underestimating.
o Example: Those who are over 65 are consistently less likely to
respond to telephone surveys, and hence are underrepresented in
telephone survey data
- 2 premises of the survey process:
o The sample describe the population
o The answers given accurately describe the respondent
characteristics
- 3 steps in data collection that produce bias:
o Choosing sample frame
o Process of selection
o Failure to collect answers from everyone selected
- Problems with Survey Wording:
o Overlapping answer options
o Double barrel questions = asking people about different things in
the same question
o Social desirability bias
This is important to avoid when you are investigating
sensitive or controversial topics
To Prevent:
Indirect questioning
Making them comfortable
Give indication that it is okay to answer in a way that is
not socially desirable
PRE-TESTING
Confidentiality/anonymity
Make the ‘idk’ option seem socially acceptable
o Leading questions = a question that is already leading to or hinting
at a desired answer
o Double negative wording = confuses people
, - Pre-testing/Piloting questions
o Its important to pre-test research instruments because surveys are
expensive to put out.
o Look at: survey wording, aesthetic design, logic and flow of wording,
clear meanings, length of time to complete survey
o How to pre-test:
Using an interview guide
Ask respondents for feedback
Cognitive interviewing = ask respondent to think aloud when
filling in the questionnaire. do they find certain questions
confusing, how easy it is.
- Question framing, AVOID:
o ambiguous terms: often, regularly, frequently
o long questions
o double-barrelled questions: there may be different answers to each
part
o very general questions, because they lack a frame of reference
o leading questions: questions that are hinting at a preferred
response
o negative terms: not, never, and double negatives make questions
unclear and confusing
o technical terms, jargon, acronyms: people need to be able to fully
understand the questions
- Common Sources of error in Surveys:
o Poorly worded question
o The way the interviewer asks it
o Interviewee misunderstands
o Interviewee misremembers
o The way the interviewer records information
o The way information is processed (coding/data entry)
- Make it easy on respondent to keep them there, consider if they are
capable/willing to respond (difficulty, memory, sensitive), and answer
categories should be exhaustive or mutually exclusive.
- Surveys have low internal validity because there is no measuring of cause
and effect.
- Surveys have high external validity when the sample is large and random.
Answer Options & Questioning
- Good questions are reliable (providing consistent measures in comparable
situations), and valid (answers correspond to what they intend to
measure).
- Evaluating questions:
o do the respondents have the requisite knowledge?
o if you just want a yes/no answer, have you given enough
possibilities?
o are you relying too much on the respondent’s memory?
o have you thought through whether you should include “don’t know”
or “decline” options?
, - Respondents need understanding & requisite knowledge to allow them to
answer.
- Answer options should be clear and not rely too much on respondent’s
memory.
- “I don’t know” or “Midpoint” Options:
Don’t include if you want to force an answer from them
Include if you want the truth/to make a choice only if they
really can
o You cannot assume that people who choose that option will be
randomly distributed over the factors
o It may/may not be chosen because it’s culturally sensitive or it may
vary on level of education
o Midpoints: some argue this means no option, and again you cannot
assume that those in the middle would be evenly distributed.
o If offered, people will use these options (idk, midpoint, middle
category) in order to spend less time thinking about answers. Some
respondents may find middle categories irritating, but also
sometimes the middle ground is a valid answer
- Social desirability bias:
o Desirability may vary between groups.
o To Prevent:
Indirect questioning
Making them comfortable
Give indication that it is okay to answer in a way that is not
socially desirable
Confidentiality/anonymity
Make the ‘idk’ option seem socially acceptable
- Balanced Questions If there is a positive side there is also a negative
side. Presentation of questions is important.
o it gives both the possibility to agree and the possibility to disagree
in the question.
- Primacy = when you look at a question, often the first options are
overrepresented. (visual online, mail)
- Recency = when you hear a question, often the later options are
overrepresented. (oral face to face, phone)
o Solution: If you are offering multiple answers, you can rotate them
every time so that there may be a bias at the individual level, but
you get a valid estimate overall. We decrease the bias at the
aggregate level through rotation.
- Acquiescence = if people don’t have a fixed opinion, they are more likely
to agree to a suggestion. To avoid this we combine negative and positively
worded items in a scale or statement.
- Types of Survey Questions:
o Dichotomous
o Open ended
o Multiple choice (mutually exclusive)
o Multiple choice (check any that apply)
o Rank order
o Likert scale
Lecture 1: Valid answers & Proper Questions
Survey Error & Bias
- We want to avoid error and bias.
- Sampling error: random (effects reliability of estimates, not much can be
done), systematic bias error (effects the validity, coverage error in the
sampling frame can be corrected).
- The way you pose/write a question effects the validity of the answers.
Think about where the error stems from to help you work out if it’s random
or systematic.
- Random Error (Sampling error) = how much the sample differs from the
population. We don’t know about the individual level, but overall, we can
expect a good approximation of the opinion.
o Invalid questions lead to mistakes
- Bias (systematic error) = in some systematic way the people responding
to a survey are consistently different from the target population as a
whole. Cannot give a good approximation of opinion but if you have a
systematic bias you’ll know if you're over/underestimating.
o Example: Those who are over 65 are consistently less likely to
respond to telephone surveys, and hence are underrepresented in
telephone survey data
- 2 premises of the survey process:
o The sample describe the population
o The answers given accurately describe the respondent
characteristics
- 3 steps in data collection that produce bias:
o Choosing sample frame
o Process of selection
o Failure to collect answers from everyone selected
- Problems with Survey Wording:
o Overlapping answer options
o Double barrel questions = asking people about different things in
the same question
o Social desirability bias
This is important to avoid when you are investigating
sensitive or controversial topics
To Prevent:
Indirect questioning
Making them comfortable
Give indication that it is okay to answer in a way that is
not socially desirable
PRE-TESTING
Confidentiality/anonymity
Make the ‘idk’ option seem socially acceptable
o Leading questions = a question that is already leading to or hinting
at a desired answer
o Double negative wording = confuses people
, - Pre-testing/Piloting questions
o Its important to pre-test research instruments because surveys are
expensive to put out.
o Look at: survey wording, aesthetic design, logic and flow of wording,
clear meanings, length of time to complete survey
o How to pre-test:
Using an interview guide
Ask respondents for feedback
Cognitive interviewing = ask respondent to think aloud when
filling in the questionnaire. do they find certain questions
confusing, how easy it is.
- Question framing, AVOID:
o ambiguous terms: often, regularly, frequently
o long questions
o double-barrelled questions: there may be different answers to each
part
o very general questions, because they lack a frame of reference
o leading questions: questions that are hinting at a preferred
response
o negative terms: not, never, and double negatives make questions
unclear and confusing
o technical terms, jargon, acronyms: people need to be able to fully
understand the questions
- Common Sources of error in Surveys:
o Poorly worded question
o The way the interviewer asks it
o Interviewee misunderstands
o Interviewee misremembers
o The way the interviewer records information
o The way information is processed (coding/data entry)
- Make it easy on respondent to keep them there, consider if they are
capable/willing to respond (difficulty, memory, sensitive), and answer
categories should be exhaustive or mutually exclusive.
- Surveys have low internal validity because there is no measuring of cause
and effect.
- Surveys have high external validity when the sample is large and random.
Answer Options & Questioning
- Good questions are reliable (providing consistent measures in comparable
situations), and valid (answers correspond to what they intend to
measure).
- Evaluating questions:
o do the respondents have the requisite knowledge?
o if you just want a yes/no answer, have you given enough
possibilities?
o are you relying too much on the respondent’s memory?
o have you thought through whether you should include “don’t know”
or “decline” options?
, - Respondents need understanding & requisite knowledge to allow them to
answer.
- Answer options should be clear and not rely too much on respondent’s
memory.
- “I don’t know” or “Midpoint” Options:
Don’t include if you want to force an answer from them
Include if you want the truth/to make a choice only if they
really can
o You cannot assume that people who choose that option will be
randomly distributed over the factors
o It may/may not be chosen because it’s culturally sensitive or it may
vary on level of education
o Midpoints: some argue this means no option, and again you cannot
assume that those in the middle would be evenly distributed.
o If offered, people will use these options (idk, midpoint, middle
category) in order to spend less time thinking about answers. Some
respondents may find middle categories irritating, but also
sometimes the middle ground is a valid answer
- Social desirability bias:
o Desirability may vary between groups.
o To Prevent:
Indirect questioning
Making them comfortable
Give indication that it is okay to answer in a way that is not
socially desirable
Confidentiality/anonymity
Make the ‘idk’ option seem socially acceptable
- Balanced Questions If there is a positive side there is also a negative
side. Presentation of questions is important.
o it gives both the possibility to agree and the possibility to disagree
in the question.
- Primacy = when you look at a question, often the first options are
overrepresented. (visual online, mail)
- Recency = when you hear a question, often the later options are
overrepresented. (oral face to face, phone)
o Solution: If you are offering multiple answers, you can rotate them
every time so that there may be a bias at the individual level, but
you get a valid estimate overall. We decrease the bias at the
aggregate level through rotation.
- Acquiescence = if people don’t have a fixed opinion, they are more likely
to agree to a suggestion. To avoid this we combine negative and positively
worded items in a scale or statement.
- Types of Survey Questions:
o Dichotomous
o Open ended
o Multiple choice (mutually exclusive)
o Multiple choice (check any that apply)
o Rank order
o Likert scale
