Chapter 2 Types of error in surveys
Surveys are designed to produce statistics about a target population. The process by
which this is done rests on inferring the characteristics of the target population from the an-
swers provided by a sample of respondents. One fundamental premise of the survey
process is that by describing the sample of people who actually respond, one can describe
the target population. A second premise is that the answers people give can be used to
accurately describe characteristics of the respondents.
Error associated with who answers
If a population consists of 50% males and 50% females, any particular sample may by
chance have more or fewer females than one would expect from the population as a
whole. In a sample survey, we usually have a single sample from which to generalize. One
goal of survey methodology is to minimize the random differences between the sample
and the population. Sampling error possibly occurs because data are collected from a
sample rather than from every single member of the population. Sampling error is random
error. A second error is bias, in some systematic way the people responding to a survey
are different from the target population as a whole. Bias to the sample can be introduced
during the sample frame, the process of selecting or when failing to collect answers from
everyone. Bias error is systematic error.
Error associated with answers
In theory, one could divide what surveys try to measure into two categories: objective facts
and subjective states. The answer given by an individual contains the true value for the
individual and the error in this answer. The extent that answers are affected by factors
other than the facts on which the answer should be based, is error in the answer. Validity
is an estimate of how well answers reflect the construct they are designed to measure.
Chapter 6 Designing questions to be good measures
Increasing the reliability of answers
Good questions are reliable (providing consistent measures in comparable situations) and
valid (answers correspond to what they are intended to measure). The answer is valuable
to the extent that it can be shown to have a predictable relationship to facts or subjective
states that are of interest. Good questions maximize the relationship between the answers
recorded and what the researcher is trying to measure. To consistent measure things each
respondent is asked the same set of questions. A good question has the following proper-
ties; the researcher’s side of question-and-answer process is scripted, the question means
the same thing to every respondent, the kinds of answers that constitute an appropriate re-
sponse to the question are communicated consistently to all respondents. Consistency of
questions can be threatened by incomplete wording, poorly defined terms, multiple ques-
tions asked at the same time, the don’t know option, difference in subgroups (e.g. lan-
guage).
Types of measures/types of questions
The extent to which the answer given is a true measure and means what the researcher
wants or expects it to mean is called validity. The idea of validity is somewhat different for
objective and subjective measures. Validity can be consistent for objective situations, for
subjective states there is no objective wau of validating the answers. Levels of
measurements are nominal, ordinal, interval and ratio. There are two groups of questions,
closed and open questions. Open questions permit the researcher to obtain answers that
more closely describe the real views of the respondents. Closed questions are usually a
more satisfactory way of creating data. The answers are more reliable, interpreting the
meaning of answers is more reliable, answer options increase the likelihood that there will
1
Surveys are designed to produce statistics about a target population. The process by
which this is done rests on inferring the characteristics of the target population from the an-
swers provided by a sample of respondents. One fundamental premise of the survey
process is that by describing the sample of people who actually respond, one can describe
the target population. A second premise is that the answers people give can be used to
accurately describe characteristics of the respondents.
Error associated with who answers
If a population consists of 50% males and 50% females, any particular sample may by
chance have more or fewer females than one would expect from the population as a
whole. In a sample survey, we usually have a single sample from which to generalize. One
goal of survey methodology is to minimize the random differences between the sample
and the population. Sampling error possibly occurs because data are collected from a
sample rather than from every single member of the population. Sampling error is random
error. A second error is bias, in some systematic way the people responding to a survey
are different from the target population as a whole. Bias to the sample can be introduced
during the sample frame, the process of selecting or when failing to collect answers from
everyone. Bias error is systematic error.
Error associated with answers
In theory, one could divide what surveys try to measure into two categories: objective facts
and subjective states. The answer given by an individual contains the true value for the
individual and the error in this answer. The extent that answers are affected by factors
other than the facts on which the answer should be based, is error in the answer. Validity
is an estimate of how well answers reflect the construct they are designed to measure.
Chapter 6 Designing questions to be good measures
Increasing the reliability of answers
Good questions are reliable (providing consistent measures in comparable situations) and
valid (answers correspond to what they are intended to measure). The answer is valuable
to the extent that it can be shown to have a predictable relationship to facts or subjective
states that are of interest. Good questions maximize the relationship between the answers
recorded and what the researcher is trying to measure. To consistent measure things each
respondent is asked the same set of questions. A good question has the following proper-
ties; the researcher’s side of question-and-answer process is scripted, the question means
the same thing to every respondent, the kinds of answers that constitute an appropriate re-
sponse to the question are communicated consistently to all respondents. Consistency of
questions can be threatened by incomplete wording, poorly defined terms, multiple ques-
tions asked at the same time, the don’t know option, difference in subgroups (e.g. lan-
guage).
Types of measures/types of questions
The extent to which the answer given is a true measure and means what the researcher
wants or expects it to mean is called validity. The idea of validity is somewhat different for
objective and subjective measures. Validity can be consistent for objective situations, for
subjective states there is no objective wau of validating the answers. Levels of
measurements are nominal, ordinal, interval and ratio. There are two groups of questions,
closed and open questions. Open questions permit the researcher to obtain answers that
more closely describe the real views of the respondents. Closed questions are usually a
more satisfactory way of creating data. The answers are more reliable, interpreting the
meaning of answers is more reliable, answer options increase the likelihood that there will
1
