Week 1 – Hoorcollege 1: Valid questions?
Proper answers
Hoofdstuk 2, 6 en 7
Introduction + Error in surveys
Survey: Inferring the characteristics of the target population from the answers provided by a
sample of respondents.
à Minimize error
à Measure the error
Premises/Sources of error:
1. By describing the sample of people who actually respond, one can describe the
target population
2. The answers people give can be used to accurately describe the characteristics of the
respondents
The design of the survey and the way data collection is carried out can affect one or both of
these potential sources of error.
Types of error
Random error Biased (systematic error)
- Sample: sampling error - Sample bias
It happens, we we’re just unlucky First row students?
- Questions: invalidity - Questions: bias
‘honest mistake’ (not remembering Did you read the literature for
exactly but trying to make a best today?
guess)
Sample frame: Those who actually have a chance to be selected
,Asking questions
Good questions are reliable and valid.
- Question reliability can be increased by a good measure
Good question properties
- The researcher’s side of the question-and-answer process is entirely scripted, so that
the questions as written fully prepare a respondent to answer.
- The question means the same thing to every respondent.
- The kinds of answers that constitute an appropriate response to the question are
communicated consistently to all respondents.
E.g.
- Double barreled questions
- Biased questions
Common sources of error in surveys
- A poorly worded question;
• Inadequate wording
• Incomplete wording
• Unacceptable optional wording
- Not ensuring consistent meaning to all respondents;
• Poorly defined terms (the general approach of avoiding complex or
ambiguous terms, and defining those that are used in the question wording,
is the best approach for most surveys)
- The way the question is asked by the interviewer;
• Multiple questions
- Misunderstanding on the part of the interviewee;
- Memory problems on the part of the interviewee;
- The way the information is recorded by the interviewer;
- The way the information is processed, either when answers are coded or when data
are entered into the computer
A simple question
- Do respondents have the requisite knowledge?
- If you just want a yes/no answer, have you given more possibilities?
- Have you thought whether you should include ‘don’t know’ options?
- Are you relying too much on the respondent’s memory?
Designing questions: general rules
- Remember your research question
- Decide exactly what you want to find out
- Consider your audience
- Imagine yourself as a respondent
o Would you answer the questions?
o How would you answer the questions?
o Identify any vague or misleading questions
, Don’t know option / Midpoint
- Je hoeft niet dezelfde distributie van antwoorden te hebben als je de DK option wel
include en als je die niet include.
- Zelfde geldt voor het midden-antwoord (als die er niet is wordt het niet per se
gelijkmatig over de andere antwoorden verdeeld)
Who doesn’t know?
- Saying “I don’t know” may be culturally sensitive
- Saying “I don’t know” varies over e.g., level of education
Should you use midpoint or don’t know (DK) option?
- If it’s offered, people will use it
o If you include DK option, people more inclined to use it
o (Some) will spend less time thinking about (tough) questions
- Same holds for middle category
(‘neutral’, ‘agree nor disagree’)
- If you want people to make a choice: leave out
- If you want people to make a choice only if they really can: include
- Beware:
o leaving out DK/middle option may cause irritation with respondent
o and sometimes the middle ground is a very viable position
Answer options
Answer options and measurement levels
- Nominaal: People or events are sorted into unordered
categories.
- Ordinaal: People or events are ordered or placed in
ordered categories along a single dimension.
o When great precision is not required
- Interval: Numbers are attached that provide
meaningful information about the distance between
ordered stimuli or classes.
- Ratio: Numbers are assigned such that ratios between
values are meaningful, as well as the intervals between them.
4 Reasons why respondents report events with less than perfect accuracy
1. They do not understand the question
2. They do not know the answer
3. They cannot recall it, although they do know it
4. They do not want to report the answer in the interview context
Being specific
- More precise is more burdensome
Proper answers
Hoofdstuk 2, 6 en 7
Introduction + Error in surveys
Survey: Inferring the characteristics of the target population from the answers provided by a
sample of respondents.
à Minimize error
à Measure the error
Premises/Sources of error:
1. By describing the sample of people who actually respond, one can describe the
target population
2. The answers people give can be used to accurately describe the characteristics of the
respondents
The design of the survey and the way data collection is carried out can affect one or both of
these potential sources of error.
Types of error
Random error Biased (systematic error)
- Sample: sampling error - Sample bias
It happens, we we’re just unlucky First row students?
- Questions: invalidity - Questions: bias
‘honest mistake’ (not remembering Did you read the literature for
exactly but trying to make a best today?
guess)
Sample frame: Those who actually have a chance to be selected
,Asking questions
Good questions are reliable and valid.
- Question reliability can be increased by a good measure
Good question properties
- The researcher’s side of the question-and-answer process is entirely scripted, so that
the questions as written fully prepare a respondent to answer.
- The question means the same thing to every respondent.
- The kinds of answers that constitute an appropriate response to the question are
communicated consistently to all respondents.
E.g.
- Double barreled questions
- Biased questions
Common sources of error in surveys
- A poorly worded question;
• Inadequate wording
• Incomplete wording
• Unacceptable optional wording
- Not ensuring consistent meaning to all respondents;
• Poorly defined terms (the general approach of avoiding complex or
ambiguous terms, and defining those that are used in the question wording,
is the best approach for most surveys)
- The way the question is asked by the interviewer;
• Multiple questions
- Misunderstanding on the part of the interviewee;
- Memory problems on the part of the interviewee;
- The way the information is recorded by the interviewer;
- The way the information is processed, either when answers are coded or when data
are entered into the computer
A simple question
- Do respondents have the requisite knowledge?
- If you just want a yes/no answer, have you given more possibilities?
- Have you thought whether you should include ‘don’t know’ options?
- Are you relying too much on the respondent’s memory?
Designing questions: general rules
- Remember your research question
- Decide exactly what you want to find out
- Consider your audience
- Imagine yourself as a respondent
o Would you answer the questions?
o How would you answer the questions?
o Identify any vague or misleading questions
, Don’t know option / Midpoint
- Je hoeft niet dezelfde distributie van antwoorden te hebben als je de DK option wel
include en als je die niet include.
- Zelfde geldt voor het midden-antwoord (als die er niet is wordt het niet per se
gelijkmatig over de andere antwoorden verdeeld)
Who doesn’t know?
- Saying “I don’t know” may be culturally sensitive
- Saying “I don’t know” varies over e.g., level of education
Should you use midpoint or don’t know (DK) option?
- If it’s offered, people will use it
o If you include DK option, people more inclined to use it
o (Some) will spend less time thinking about (tough) questions
- Same holds for middle category
(‘neutral’, ‘agree nor disagree’)
- If you want people to make a choice: leave out
- If you want people to make a choice only if they really can: include
- Beware:
o leaving out DK/middle option may cause irritation with respondent
o and sometimes the middle ground is a very viable position
Answer options
Answer options and measurement levels
- Nominaal: People or events are sorted into unordered
categories.
- Ordinaal: People or events are ordered or placed in
ordered categories along a single dimension.
o When great precision is not required
- Interval: Numbers are attached that provide
meaningful information about the distance between
ordered stimuli or classes.
- Ratio: Numbers are assigned such that ratios between
values are meaningful, as well as the intervals between them.
4 Reasons why respondents report events with less than perfect accuracy
1. They do not understand the question
2. They do not know the answer
3. They cannot recall it, although they do know it
4. They do not want to report the answer in the interview context
Being specific
- More precise is more burdensome
