WEEK 1
(Chapter 2, 6, 7)
Chapter 2 – Types of Error in Surveys
Two of the main goals of survey methodology are to
minimize error in data collected by surveys and to measure
the error that necessarily is part of any survey.
Two fundamental premises and errors:
1. by describing the sample of people who actually
respond, ones can describe the target population
Issue: How closely sample responding mirrors
population
2. the answers people give can be used to accurately
describe characteristics of the respondents
Issue: How well answers measure characteristics to
be describes
Error associated with who answers
Sampling error (=random error)
● One kind of error concerns the random variation from the true characteristics of the
population. this variation, the possible error that stems solely from the fact that data
are collected from a sample rather than from every single member of the population
is called sampling error. (e.g. by chance too many females in sample)
Bias (=systematic error)
● A second kind of error that effect the relationship between a sample and the
population is bias. Bias means hat in some systematic way the people responding to
a survey are different from the target population as a whole.
three potential threats to introduce bias in the data collection process:
1. sample frame
Does everyone have the same chance of being selected? (USA: most surveys leave
out people in prisons and nursing homes)
2. process of selecting
if the selection is not random, the results could be a sample that is different from the
population as a whole. (people who volunteer are different from random selection)
3. failure to collect answers from everyone
Some people are not available to answer, some refuse, some lack language skills,
etc. The people who don’t answer are different from the people who answer
Error associated with answers
surveys try to measure two categories: (1) objective facts: accuracy of answer can be
directly assessed, e.g. height, employment, etc. and (2) subjective facts: no objective way to
verify: e.g. how tired someone feels
1
,Validity
Respondents’ estimates of how tired they have been over the past week may be affected by
how tired they feel at the time they are answering the questions. The point is that to the
extent that answers are affected by factors other than the facts on which the answer should
be based, there is error in the answer. Validity is the term that psychologists use to describe
the relationship between an answer and some measure of the true score
Recap of Errors in Surveys
Chapter 6 – Designing Questions to Be Good Measures
Good questions are reliable (providing consistent measures in comparable situations) and
valid (answers correspond to what they are intended to measure).
Increasing the reliability of answers
In order to provide a consistent data collection experience for all respondents, a good
question has the following properties:
● The researcher’s side of the question-and-answer process is entirely scripted, so that
the questions as written fully prepare a respondent to answer.
● The question means the same thing to every respondent.
● The kinds of answers that constitute an appropriate response to the question are
communicated consistently to all respondents.
Inadequate wording
The simplest example of inadequate question wording is when, somehow, the researcher’s
words do not constitute a complete question.
Incomplete wording
Bad: Age? Better: What was your age on your last birthday?
Optional wording
Q: Did (you/he/she) report the attack to the police?
The interviewers has to make a choice: use the right pronoun
Unacceptable optional wording
Q: What do you like best about this neighborhood? (We’re interested in anything such as
houses, the people, the parks, or whatever.)
If interviewers use the parenthetical probe when a respondent does not readily come up with
an answer, that subset of respondents will have answered a different question.
2
,→ undermines the principle of standardized interviewing
Poor wording
Q: I would like you to rate different features of your neighborhood as very good, good, fair,
or poor. Please think carefully about each item as I read it.
a. Public schools
b. Parks
c. Public transportation
d. Other
The response alternatives are read prior to an instruction to think carefully about the specific
items. The respondent probably will forget the question.
Better wording
Q: I am going to ask you to rate different features of your neighborhood. I want you to think
carefully about your answers. How would you rate (FEATURE)—would you say very
good, good, fair, or poor?
This format gives the interviewer the wording needed for asking the first and all subsequent
items on the list as complete questions. But problem: other? how to ask for that?
Ensuring consistent meaning to all respondents
The questions should all mean the same to all respondents. Problems that might evoke are:
(1) using words that are not understood universally, and (2) using terms or concepts that can
have multiple meanings.
Poorly defined terms
Q:How many times in the past year have you seen or talked with a doctor about your health?
Problem: what is considered a doctor? what constitutes seeing or talking with a doctor?
Solution: provide definition
Q: Did you eat breakfast yesterday?
Problem: what is breakfast? only coffee and donut? only consumed before 8 a.m?
Solution: ask what has been consumed before 10 a.m
Avoiding multiple questions
Q: Do you want to be rich and famous?
Problem: rich and famous are not the same
Q: In the last 30 days, when you withdrew cash from an ATM machine, how often did you
withdraw less than $25—always, usually, sometimes, never?
Problem: need three cognitive calculations (how often going to ATM, …)
Solution: split up question
1. In the last 30 days, how many times did you withdraw cash from an ATM machine?
2. (IF ANY) On how many of those times did you withdraw less than $25?
Q: To what kind of place do you go for your routine medical care?
Problem: question assumes that everyone gets routine medical care
Solution: first ask if, then proceed
3
, The “Don’t Know” option
Problem: do you have the information needed to answer the question and, if so, what is the
answer? The alternative is to ask all respondents a standardized screening question about
whether or not they have the information needed to answer a question.
→ refer to lectures
Specialized Wording for Special Subgroups
One could argue that standardized measurement actually would require different questions
for different subgroups. However, methodologists tend to work very hard to attempt to find
wording for questions that has consistent meaning across an entire population. The extreme
challenge is how to collect comparable data from people who speak different languages.
Standardized Expectations for Type of Response
In addition to giving interviewers a good script so that they can read the questions exactly as
worded and designing questions that mean the same thing to all respondents,the other key
component of a good question is that respondents have the same perception of what
constitutes an adequate answer for the question.
Simple way: go with closed questions, BUT do not always suitable
Q: When did you have pneumonia?
Problem: “Five years ago”, “In 1987”, “When I was 32”
Solution: How old were you when you had pneumonia
→ always specify what exact information you want to know to get consistent answers
Types of Measures/ Types of Questions
The extent to which the answer given is a true measure and means what the researcher
wants or expects it to mean is called validity.
● questions that measure facts/objective measurable events: it is possible to check
the accuracy of an answer by some independent observation
● questions that ask about subjective states: no objective way of validating the
answers
Levels of measurement
1. Nominal: People or events are sorted into unordered categories. (Are you male or
female?)
2. Ordinal: People or events are ordered or placed in ordered categories along a single
dimension.(How would you rate your health: very good, good, fair, or poor?)
3. Interval data: Numbers are attached that provide meaningful information about the
distance between ordered stimuli or classes (in fact, interval data are very rare;
Fahrenheit temperature is one of the few common examples).
4. Ratio data: Numbers are assigned such that ratios between values are meaningful,
as well as the intervals between them. Common examples are counts or
measurements by an objective, physical scale, such as distance, weight, or time.
(How old were you on your last birthday?)
Most often in surveys, when one is collecting factual data, respondents are asked to fit
themselves or their experiences into a category, creating nominal data (Are you married?),
4
(Chapter 2, 6, 7)
Chapter 2 – Types of Error in Surveys
Two of the main goals of survey methodology are to
minimize error in data collected by surveys and to measure
the error that necessarily is part of any survey.
Two fundamental premises and errors:
1. by describing the sample of people who actually
respond, ones can describe the target population
Issue: How closely sample responding mirrors
population
2. the answers people give can be used to accurately
describe characteristics of the respondents
Issue: How well answers measure characteristics to
be describes
Error associated with who answers
Sampling error (=random error)
● One kind of error concerns the random variation from the true characteristics of the
population. this variation, the possible error that stems solely from the fact that data
are collected from a sample rather than from every single member of the population
is called sampling error. (e.g. by chance too many females in sample)
Bias (=systematic error)
● A second kind of error that effect the relationship between a sample and the
population is bias. Bias means hat in some systematic way the people responding to
a survey are different from the target population as a whole.
three potential threats to introduce bias in the data collection process:
1. sample frame
Does everyone have the same chance of being selected? (USA: most surveys leave
out people in prisons and nursing homes)
2. process of selecting
if the selection is not random, the results could be a sample that is different from the
population as a whole. (people who volunteer are different from random selection)
3. failure to collect answers from everyone
Some people are not available to answer, some refuse, some lack language skills,
etc. The people who don’t answer are different from the people who answer
Error associated with answers
surveys try to measure two categories: (1) objective facts: accuracy of answer can be
directly assessed, e.g. height, employment, etc. and (2) subjective facts: no objective way to
verify: e.g. how tired someone feels
1
,Validity
Respondents’ estimates of how tired they have been over the past week may be affected by
how tired they feel at the time they are answering the questions. The point is that to the
extent that answers are affected by factors other than the facts on which the answer should
be based, there is error in the answer. Validity is the term that psychologists use to describe
the relationship between an answer and some measure of the true score
Recap of Errors in Surveys
Chapter 6 – Designing Questions to Be Good Measures
Good questions are reliable (providing consistent measures in comparable situations) and
valid (answers correspond to what they are intended to measure).
Increasing the reliability of answers
In order to provide a consistent data collection experience for all respondents, a good
question has the following properties:
● The researcher’s side of the question-and-answer process is entirely scripted, so that
the questions as written fully prepare a respondent to answer.
● The question means the same thing to every respondent.
● The kinds of answers that constitute an appropriate response to the question are
communicated consistently to all respondents.
Inadequate wording
The simplest example of inadequate question wording is when, somehow, the researcher’s
words do not constitute a complete question.
Incomplete wording
Bad: Age? Better: What was your age on your last birthday?
Optional wording
Q: Did (you/he/she) report the attack to the police?
The interviewers has to make a choice: use the right pronoun
Unacceptable optional wording
Q: What do you like best about this neighborhood? (We’re interested in anything such as
houses, the people, the parks, or whatever.)
If interviewers use the parenthetical probe when a respondent does not readily come up with
an answer, that subset of respondents will have answered a different question.
2
,→ undermines the principle of standardized interviewing
Poor wording
Q: I would like you to rate different features of your neighborhood as very good, good, fair,
or poor. Please think carefully about each item as I read it.
a. Public schools
b. Parks
c. Public transportation
d. Other
The response alternatives are read prior to an instruction to think carefully about the specific
items. The respondent probably will forget the question.
Better wording
Q: I am going to ask you to rate different features of your neighborhood. I want you to think
carefully about your answers. How would you rate (FEATURE)—would you say very
good, good, fair, or poor?
This format gives the interviewer the wording needed for asking the first and all subsequent
items on the list as complete questions. But problem: other? how to ask for that?
Ensuring consistent meaning to all respondents
The questions should all mean the same to all respondents. Problems that might evoke are:
(1) using words that are not understood universally, and (2) using terms or concepts that can
have multiple meanings.
Poorly defined terms
Q:How many times in the past year have you seen or talked with a doctor about your health?
Problem: what is considered a doctor? what constitutes seeing or talking with a doctor?
Solution: provide definition
Q: Did you eat breakfast yesterday?
Problem: what is breakfast? only coffee and donut? only consumed before 8 a.m?
Solution: ask what has been consumed before 10 a.m
Avoiding multiple questions
Q: Do you want to be rich and famous?
Problem: rich and famous are not the same
Q: In the last 30 days, when you withdrew cash from an ATM machine, how often did you
withdraw less than $25—always, usually, sometimes, never?
Problem: need three cognitive calculations (how often going to ATM, …)
Solution: split up question
1. In the last 30 days, how many times did you withdraw cash from an ATM machine?
2. (IF ANY) On how many of those times did you withdraw less than $25?
Q: To what kind of place do you go for your routine medical care?
Problem: question assumes that everyone gets routine medical care
Solution: first ask if, then proceed
3
, The “Don’t Know” option
Problem: do you have the information needed to answer the question and, if so, what is the
answer? The alternative is to ask all respondents a standardized screening question about
whether or not they have the information needed to answer a question.
→ refer to lectures
Specialized Wording for Special Subgroups
One could argue that standardized measurement actually would require different questions
for different subgroups. However, methodologists tend to work very hard to attempt to find
wording for questions that has consistent meaning across an entire population. The extreme
challenge is how to collect comparable data from people who speak different languages.
Standardized Expectations for Type of Response
In addition to giving interviewers a good script so that they can read the questions exactly as
worded and designing questions that mean the same thing to all respondents,the other key
component of a good question is that respondents have the same perception of what
constitutes an adequate answer for the question.
Simple way: go with closed questions, BUT do not always suitable
Q: When did you have pneumonia?
Problem: “Five years ago”, “In 1987”, “When I was 32”
Solution: How old were you when you had pneumonia
→ always specify what exact information you want to know to get consistent answers
Types of Measures/ Types of Questions
The extent to which the answer given is a true measure and means what the researcher
wants or expects it to mean is called validity.
● questions that measure facts/objective measurable events: it is possible to check
the accuracy of an answer by some independent observation
● questions that ask about subjective states: no objective way of validating the
answers
Levels of measurement
1. Nominal: People or events are sorted into unordered categories. (Are you male or
female?)
2. Ordinal: People or events are ordered or placed in ordered categories along a single
dimension.(How would you rate your health: very good, good, fair, or poor?)
3. Interval data: Numbers are attached that provide meaningful information about the
distance between ordered stimuli or classes (in fact, interval data are very rare;
Fahrenheit temperature is one of the few common examples).
4. Ratio data: Numbers are assigned such that ratios between values are meaningful,
as well as the intervals between them. Common examples are counts or
measurements by an objective, physical scale, such as distance, weight, or time.
(How old were you on your last birthday?)
Most often in surveys, when one is collecting factual data, respondents are asked to fit
themselves or their experiences into a category, creating nominal data (Are you married?),
4
