Samenvatting: Survey research methods – Floyd Fowler
Chapter 2: types of error in a survey
One fundamental premise of the survey research process is that by describing the sample of
people who actually respond one can describe the target population
A second fundamental premise of survey research is that the answers people give can be used
to accurately describe characteristics of the respondents
So you want: to make good inferences
Well answers to measure characteristics
A sample responding who mirrors the population
The above are potential sources of error, the design and the way the data is collected can
affect both of one of these.
Error associated with who answers
A sample can always differ from the larger population. One goal of survey methodology is to
minimize the random differences between the sample and the population, but on average it
evens out.
Sampling error = (=random error) random variation from the true characteristics of the
population can lead to possible error because you collect data from a sample and not the
whole population
Bias = (second kind of error = systematic) in some kind of way the people responding to a
survey are different from the target population as a whole
3 steps in process of collecting data that can affect bias:
1. Choosing a sample frame: those who actually have a chance to be selected. For
variables on which people who are included vs Those who are systematically left out,
the data will be biased (for example: all of those without a phone, when that’s your
method)
2. Process of selecting: has to be random in order to not have bias
3. Failure to collect answers from everyone selected to be in the sample is a potential
source of bias: unable to answer, not available (health, language), refuse etc. this
affects the answers and result may be biased.
Error associated with answers
What surveys try to measure can be divided in two categories: objective facts (voted or not,
height etc.) you can directly asses how accurate the answers where) or subjective facts
psychometricians think of answers consisting of two components: the true score/value for
individual + error in the answer given by individual
Errors can be caused by all kinds of things: misunderstanding, lack of knowledge etc.
rounding up or down your height, etc.
Validity = the relationship between an answer and some measure of the true score/degree to
which answers systematically differ from a true score in one direction. So you want to make
the error term as small as possible so the answers mainly reflect the true score. If the error
associated with answers is random the average answers should be the same as the average
true value.
, For questions designed for objective facts, they have a potential to be biased, like how many
cigarettes one smokes, estimates of these behaviours are likely to be biased systematically
different from the true scores.
The idea of validity of subjective measures can’t be observed directly. The calculations are
more complicated, but the end result is the same: an estimate of how well answers reflect the
construct they are designed to measure.
So critical for the quality of the survey you want:
- Inference that answers can be used to accurately describe a sample of respondents: so
generalization from answers to reality
- And accurately generalize from a sample of respondents to an entire population
Two kinds of error:
1. Random error: variability around the true values
2. Systematic error: (biased) differences between sample and population OR between
answers and true values
Chapter 3: sampling
Sometimes the goal of information gathering is not to generate statistis about a population
but describe a set of people in more general way for these purposes available or volunteers
may be useful. The way to evaluate a sample is by examining the process by which it was
selected. 3 key aspects of sample selection:
1. How well sample frame corresponds to the population a researcher wants to describe
sample frame = people who have a chance to be selected
2. Probability sampling each person must have a know chance of selection, if
researcher of respondent (e.g. availability) affect these chances there is no statistical
basis for evaluating the sample representativeness
3. The details of sample design (size, procedures) will influence precision of estimates
The sample frame: most of the times is done in one of these ways:
1. Done from more or less complete list of individuals in population
2. Done from a set of people who go somewhere or do something that enables them to be
sampled (no advanced list)
3. Addresses or housing units are sampled as first stage of selecting a sample: can be
done from a list, geographic areas or numbers associated with housing units
Characteristics of a sample from you should evaluate:
1. Comprehensiveness: how completely it covers the target population: general list are
often very exclusive (registered voters e.g.), published telephones often covers only
50% of population, households exclude homeless, prisoners etc.
2. Whether or not a person’s probability of selection can be calculated
3. Efficiency, or rate at which target population can be found among those in frame. In
some cases, sampling frame includes people not part of target population
Selecting a one-stage sample
Simple random sampling requires numbered list of population, members are selected
once at a time
Systematic sample: mechanically easier to create than above. The benefits of
stratification can be accomplished more easily.
Chapter 2: types of error in a survey
One fundamental premise of the survey research process is that by describing the sample of
people who actually respond one can describe the target population
A second fundamental premise of survey research is that the answers people give can be used
to accurately describe characteristics of the respondents
So you want: to make good inferences
Well answers to measure characteristics
A sample responding who mirrors the population
The above are potential sources of error, the design and the way the data is collected can
affect both of one of these.
Error associated with who answers
A sample can always differ from the larger population. One goal of survey methodology is to
minimize the random differences between the sample and the population, but on average it
evens out.
Sampling error = (=random error) random variation from the true characteristics of the
population can lead to possible error because you collect data from a sample and not the
whole population
Bias = (second kind of error = systematic) in some kind of way the people responding to a
survey are different from the target population as a whole
3 steps in process of collecting data that can affect bias:
1. Choosing a sample frame: those who actually have a chance to be selected. For
variables on which people who are included vs Those who are systematically left out,
the data will be biased (for example: all of those without a phone, when that’s your
method)
2. Process of selecting: has to be random in order to not have bias
3. Failure to collect answers from everyone selected to be in the sample is a potential
source of bias: unable to answer, not available (health, language), refuse etc. this
affects the answers and result may be biased.
Error associated with answers
What surveys try to measure can be divided in two categories: objective facts (voted or not,
height etc.) you can directly asses how accurate the answers where) or subjective facts
psychometricians think of answers consisting of two components: the true score/value for
individual + error in the answer given by individual
Errors can be caused by all kinds of things: misunderstanding, lack of knowledge etc.
rounding up or down your height, etc.
Validity = the relationship between an answer and some measure of the true score/degree to
which answers systematically differ from a true score in one direction. So you want to make
the error term as small as possible so the answers mainly reflect the true score. If the error
associated with answers is random the average answers should be the same as the average
true value.
, For questions designed for objective facts, they have a potential to be biased, like how many
cigarettes one smokes, estimates of these behaviours are likely to be biased systematically
different from the true scores.
The idea of validity of subjective measures can’t be observed directly. The calculations are
more complicated, but the end result is the same: an estimate of how well answers reflect the
construct they are designed to measure.
So critical for the quality of the survey you want:
- Inference that answers can be used to accurately describe a sample of respondents: so
generalization from answers to reality
- And accurately generalize from a sample of respondents to an entire population
Two kinds of error:
1. Random error: variability around the true values
2. Systematic error: (biased) differences between sample and population OR between
answers and true values
Chapter 3: sampling
Sometimes the goal of information gathering is not to generate statistis about a population
but describe a set of people in more general way for these purposes available or volunteers
may be useful. The way to evaluate a sample is by examining the process by which it was
selected. 3 key aspects of sample selection:
1. How well sample frame corresponds to the population a researcher wants to describe
sample frame = people who have a chance to be selected
2. Probability sampling each person must have a know chance of selection, if
researcher of respondent (e.g. availability) affect these chances there is no statistical
basis for evaluating the sample representativeness
3. The details of sample design (size, procedures) will influence precision of estimates
The sample frame: most of the times is done in one of these ways:
1. Done from more or less complete list of individuals in population
2. Done from a set of people who go somewhere or do something that enables them to be
sampled (no advanced list)
3. Addresses or housing units are sampled as first stage of selecting a sample: can be
done from a list, geographic areas or numbers associated with housing units
Characteristics of a sample from you should evaluate:
1. Comprehensiveness: how completely it covers the target population: general list are
often very exclusive (registered voters e.g.), published telephones often covers only
50% of population, households exclude homeless, prisoners etc.
2. Whether or not a person’s probability of selection can be calculated
3. Efficiency, or rate at which target population can be found among those in frame. In
some cases, sampling frame includes people not part of target population
Selecting a one-stage sample
Simple random sampling requires numbered list of population, members are selected
once at a time
Systematic sample: mechanically easier to create than above. The benefits of
stratification can be accomplished more easily.
