Week 1
Survey: set statistical estimates/draw inferences (about a population) of characteristics of a targeted
population
Two main goals of survey:
1. Minimize error in data collected (minimize variability of a sample, for its characteristic to be similar to
the population)
2. Measure the error that is part of the survey
Survey:
Advantages →
1. Low cost
2. Fast delivery
3. Rapid data processing
4. Can research large population
5. Various modes (through phone, email…)
Disadvantages →
1. Limited to scale or checklist questions
2. No control over response rate
3. Provides information more than understanding
4. Increasing public resistance (people deciding not to participate)
5. Difficult to explore issues in depth
Premises/Inferences:
1. By describing the sample of people who actually respond, one can describe the targeted
population
- Characteristics the survey is designed to describe are present to the same degree and are
distributed in the same way, in the sample responding as in the target population
2. Answers people give can be used to accurately describe the characteristics of respondents
Issues are sources of error
-Describing sample of respondents → accurately describing the targeted population
Error associated with WHO answers:
,Two types of errors:
1. Sampling/Random error- the possible error that stems solely from the fact that data are collected
from a sample rather than from every single member of the population (the difference between
the sample mean and population mean)
- Affects findings in unpredictable ways
- On a question level, it affects validity -on an individual level (how many songs did you
listen to this week)
- Reliability→ errors not predictable/ research inconsistent
- Need to minimize random variation
- “People over 65 usually speak less on the telephone so they are underrepresented in
telephone surveys”
- Solution = sample more
2. Bias/Systematic error (consistent under/over representation)- in some systematic way the
people responding to a survey are different from the target population as a whole
- “Only first row students”
- Affects in predictable ways
- Affects validity
- Having some elements of the population underrepresented in our sample of respondents
or from some systematic distortion of answers to the questions that we posed
- Bias in questions: Social desirability, Leading questions
- “No.of cigarettes smoked is consistently underrepresented in surveys”
- Solution = e.g. using controls in experiments
- Selection bias -bigger sample not always better; coverage bias (sample frame-
over/under-estimates)
Collecting data can introduce bias into the sample:
1. When choosing a sample frame- when people have no choice of being sampled (leave out people
who have no home address at all- sample will be bias)
2. If the process of selecting a sample is not random- selecting volunteers
3. Failure to collect answers from everyone selected (people refuse to answer)
Error associated with answers:
, What survey tries to measure:
1. Objective facts (height, employment)
- The way in which answers differ from the true score→ respondents underreport how many
cigarettes they smoke and how much alcohol they drink; Bias occurs when rounding numbers
2. Subjective facts (political views of a person); can’t be verified
- Cannot measure bias; Validity cannot be observed directly but how answers are related to other
similar measures
- The way to assess the answers to a question is to measure how well they correspond to the
“truth.”
Answers consist of:
1. True score (what a perfect reporter with perfect knowledge would give as an answer)
2. An element of error (misunderstanding the question, not having the information needed to answer…)
xi(answer) = ti(true value) + ei(error)
- To the extent that answers are affected by factors other than the facts on which the answer should
be based, there is error in the answer
Validity- The extent to which the answer given is a true measure and means what the researcher wants or
expects it to mean
Common sources of error in surveys:
● Poorly worded question/ ambiguous (Random error)
● Deviations from the script by the interviewer (Random error)
● Misunderstanding on the part of the respondent (Random error)
● Memory problems by respondent (Random error)
● The way information is processed by an interviewer
● The way information is processed when answers are coded
Problematic questions:
1. Double-barreled (solution: separate the questions)
2. Leading questions (possible in some cases, like in socially desirable questions)
3. Double-negative questions
4. Using ambiguous terms- Often, regularly, frequently
5. Technical terms
6. Very general questions
7. Long questions
Don’t know option/Midpoints: invalidity?
-If you put it, people will use it
-If you need people to choose one or the other: do not include don’t know
-If people suffer from lack of knowledge, don’t know option could be sensible
-Spend less time thinking about difficult questions
-Leaving out DK option may cause irritation
- Middle ground/don’t know is a possibility
Survey: set statistical estimates/draw inferences (about a population) of characteristics of a targeted
population
Two main goals of survey:
1. Minimize error in data collected (minimize variability of a sample, for its characteristic to be similar to
the population)
2. Measure the error that is part of the survey
Survey:
Advantages →
1. Low cost
2. Fast delivery
3. Rapid data processing
4. Can research large population
5. Various modes (through phone, email…)
Disadvantages →
1. Limited to scale or checklist questions
2. No control over response rate
3. Provides information more than understanding
4. Increasing public resistance (people deciding not to participate)
5. Difficult to explore issues in depth
Premises/Inferences:
1. By describing the sample of people who actually respond, one can describe the targeted
population
- Characteristics the survey is designed to describe are present to the same degree and are
distributed in the same way, in the sample responding as in the target population
2. Answers people give can be used to accurately describe the characteristics of respondents
Issues are sources of error
-Describing sample of respondents → accurately describing the targeted population
Error associated with WHO answers:
,Two types of errors:
1. Sampling/Random error- the possible error that stems solely from the fact that data are collected
from a sample rather than from every single member of the population (the difference between
the sample mean and population mean)
- Affects findings in unpredictable ways
- On a question level, it affects validity -on an individual level (how many songs did you
listen to this week)
- Reliability→ errors not predictable/ research inconsistent
- Need to minimize random variation
- “People over 65 usually speak less on the telephone so they are underrepresented in
telephone surveys”
- Solution = sample more
2. Bias/Systematic error (consistent under/over representation)- in some systematic way the
people responding to a survey are different from the target population as a whole
- “Only first row students”
- Affects in predictable ways
- Affects validity
- Having some elements of the population underrepresented in our sample of respondents
or from some systematic distortion of answers to the questions that we posed
- Bias in questions: Social desirability, Leading questions
- “No.of cigarettes smoked is consistently underrepresented in surveys”
- Solution = e.g. using controls in experiments
- Selection bias -bigger sample not always better; coverage bias (sample frame-
over/under-estimates)
Collecting data can introduce bias into the sample:
1. When choosing a sample frame- when people have no choice of being sampled (leave out people
who have no home address at all- sample will be bias)
2. If the process of selecting a sample is not random- selecting volunteers
3. Failure to collect answers from everyone selected (people refuse to answer)
Error associated with answers:
, What survey tries to measure:
1. Objective facts (height, employment)
- The way in which answers differ from the true score→ respondents underreport how many
cigarettes they smoke and how much alcohol they drink; Bias occurs when rounding numbers
2. Subjective facts (political views of a person); can’t be verified
- Cannot measure bias; Validity cannot be observed directly but how answers are related to other
similar measures
- The way to assess the answers to a question is to measure how well they correspond to the
“truth.”
Answers consist of:
1. True score (what a perfect reporter with perfect knowledge would give as an answer)
2. An element of error (misunderstanding the question, not having the information needed to answer…)
xi(answer) = ti(true value) + ei(error)
- To the extent that answers are affected by factors other than the facts on which the answer should
be based, there is error in the answer
Validity- The extent to which the answer given is a true measure and means what the researcher wants or
expects it to mean
Common sources of error in surveys:
● Poorly worded question/ ambiguous (Random error)
● Deviations from the script by the interviewer (Random error)
● Misunderstanding on the part of the respondent (Random error)
● Memory problems by respondent (Random error)
● The way information is processed by an interviewer
● The way information is processed when answers are coded
Problematic questions:
1. Double-barreled (solution: separate the questions)
2. Leading questions (possible in some cases, like in socially desirable questions)
3. Double-negative questions
4. Using ambiguous terms- Often, regularly, frequently
5. Technical terms
6. Very general questions
7. Long questions
Don’t know option/Midpoints: invalidity?
-If you put it, people will use it
-If you need people to choose one or the other: do not include don’t know
-If people suffer from lack of knowledge, don’t know option could be sensible
-Spend less time thinking about difficult questions
-Leaving out DK option may cause irritation
- Middle ground/don’t know is a possibility
