MCRS - A
WEEK 6A - SURVEYS AND SAMPLING
Surveys
A series of formatted questions to a defined sample of people with the expectation of
responses immediately or within a few days.
Not quite the same as a questionnaire, which is the actual set of questions itself.
- Capture public opinion at a point in time
- Use formatted (standardized) rather than open-ended questions
- Administered by phone, mail, email, web, or f2f interview
- With appropriate sampling, we can generalize with a known statistical confidence
from a sample to a wider population
Advantages Disadvantages
- Relatively low cost - Limited to scale / checklist questions
- Relatively fast delivery - No control over response rate
- Fast data processing - Provides more information than
- Can research large populations understanding
- Many methods - Increasing public resistance
- Hard to explore issues in depth
Types of surveys
Cross-sectional: Capture what is going on at a specific point in time (slice of life)
Asks 1 group of people (respondents) the same questions at 1 point in time
Least expensive but can’t determine causality
Example: Self-esteem and marital satisfaction are measured for 100 individuals at one
point in time.
Longitudinal: Observations taken over time more than once to track changes
Can only detect causality sometimes (see: panel)
Considers the influence of time
Trend Drawing different samples from the same population over time
Each study collects data from different individuals each time
No clear “why” because you’re not collecting information from the same
individuals
Advantage: can maintain the sample size
Disadvantage: N o assurance that the new people won’t differ from others
Example: In 2007, 1k people are asked for their support of same-sex marriage.
In 2017, a different set of 1k people are asked for their support of same-sex
marriage.
The difference in opinions is the trend in support for same-sex marriage.
, Cohort Examines changes in a cohort across time
Each study examines different people in the same cohort
Cohort: A group of people with shared characteristics
Advantage: Provides more detail than a trend study
Disadvantage: Determining causality is a problem because you are not studying
the same people over time
Example: How the political views of millennials change over time; babies
Panel Examines the same individuals over time
Can indicate causality, but not definitely determine it
Multiple “waves”, but must account for attrition
Advantage: Can provide far more in-depth understanding (can examine the key
determinants of changes)
Disadvantage: Very expensive
Example: Study aimed at examining the relationship between risky sexual
behaviour and perceptions of peer involvement
Survey mode
With interviewer Response Speed Costs Complexity
Face to face Good Slow (Very) high (Very) high
Phone Reasonable Fast High (Very) high
Without interviewer
Paper & pencil Low Slow Low/moderate Low
Email / web Low Very fast Very low Very high
Types of questions
- Filter questions: Route respondents around questions that don’t apply to them
- Example: If this home has an Internet connection, please continue with
question 6. If it has no Internet connection, please go to question 18.
- Easier with online surveys
- Dichotomous: Respondents must selection one of two options (e.g. yes / no)
- More simplified but life is not that simple
- Open-ended: Allows the respondent to answer however they want
- Get more insight but much harder to code
- Multiple choice: Choose from a few mutually exclusive options
- “Select one” vs. “select many” depends on the necessary information
- Rank order: Ranking questions or statements in order of importance
- Likert scale: Indicate the level of agreement or disagreement
- Semantic-differential scale: Position on a scale between two bipolar adjectives
,Survey wording
- Leading questions: Force the respondent into an assumption that may not be true
- E.g. How stupid is Donald Trump when it comes to immigration related
issues?
- Double-barrelled questions: Asking two questions simultaneously, but allows for
only one answer
- E.g. I felt welcomed by the lecturers and other staff on my first day at UvA
(totally disagree – totally agree)
- Framing questions: When researchers (un)willingly influence the respondent’s
answers
- Negative wording: Easily misunderstood
- E.g. a statistics course should not be a statistics requirement
- Double-negative wording: Negative wording in combination with a double-barrelled
question
- E.g.What is your stance on not allowing selling alcohol?
- Funnel design: Start with broad questions and then go specific (better with phone)
- Inverted funnel design: Start with specific question and then go broad (better
online)
Social desirability bias: A type of response bias that survey respondents answer questions
in a manner that they believe will be viewed favorably by others. The tendency poses a
serious problem with conducting research with self-reports.
To avoid this:
- Indirect questioning
- Show it is ok to answer in a way that it is not desirable (make undesirable answer
less threatening)
Research validity
- Internal validity: No other explanations for causality
- External validity: U nits of analysis are representative
Pre-testing/ pilot testing
Testing the survey first in order to filter out mistakes.
Look at:
- Survey wording
- Aesthetics and design of questionnaire
- Logic and flow of wording
- Words – and their specific meaning(s)
- Length/time it takes to complete questionnaire
How to pre-test?
1. Ask respondents to fill in the questionnaire and add evaluation questions (ask for
feedback)
2. Cognitive interviewing: use think-aloud method and ask respondents what they
think when filling in the questionnaire
3. Ask your supervisor, fellow student or colleague for feedback
, Sampling
A statistical method of obtaining representative data or observations from a population.
- Population: Every person in the group that the researcher wants to study
- Large = difficult
- Sample: Part of a population selected for study
- Hopefully representative
- Census: Study of entire population
- Parameters: Describe a population
- Statistics: Describes a sample
- Sampling frame: List from which a sample is drawn, all the units in a population
- Sampling error: Degree to which the sample statistics differ from the parameters of
the population
Probability sampling
Hopefully representative and generalizable. All units of analysis have an equal chance of
selection, and there is no need for huge samples.
Simple random Removes the researcher as an agent of selection, limiting bias and
sampling allowing for greater representation.
Stratified random Involves subdividing the market into segments (known as strata)
sampling that share homogeneous or very similar characteristics.
A number of respondents from each stratum that is proportional
to the population is then randomly selected for the sample.
Systematic sampling A random starting point followed by selecting every n
th person.
Multistage cluster The taking of samples in stages using smaller and smaller
sampling sampling units at each stage. It is a complex form of cluster
sampling.
Cluster sampling The researcher divides the population into separate groups, called
clusters. Then, a simple random sample of clusters is selected
from the population. Every member from the chosen clusters are
sampled (but not all clusters are used). Clusters can be school
classes, geographical locations etc.
WEEK 6A - SURVEYS AND SAMPLING
Surveys
A series of formatted questions to a defined sample of people with the expectation of
responses immediately or within a few days.
Not quite the same as a questionnaire, which is the actual set of questions itself.
- Capture public opinion at a point in time
- Use formatted (standardized) rather than open-ended questions
- Administered by phone, mail, email, web, or f2f interview
- With appropriate sampling, we can generalize with a known statistical confidence
from a sample to a wider population
Advantages Disadvantages
- Relatively low cost - Limited to scale / checklist questions
- Relatively fast delivery - No control over response rate
- Fast data processing - Provides more information than
- Can research large populations understanding
- Many methods - Increasing public resistance
- Hard to explore issues in depth
Types of surveys
Cross-sectional: Capture what is going on at a specific point in time (slice of life)
Asks 1 group of people (respondents) the same questions at 1 point in time
Least expensive but can’t determine causality
Example: Self-esteem and marital satisfaction are measured for 100 individuals at one
point in time.
Longitudinal: Observations taken over time more than once to track changes
Can only detect causality sometimes (see: panel)
Considers the influence of time
Trend Drawing different samples from the same population over time
Each study collects data from different individuals each time
No clear “why” because you’re not collecting information from the same
individuals
Advantage: can maintain the sample size
Disadvantage: N o assurance that the new people won’t differ from others
Example: In 2007, 1k people are asked for their support of same-sex marriage.
In 2017, a different set of 1k people are asked for their support of same-sex
marriage.
The difference in opinions is the trend in support for same-sex marriage.
, Cohort Examines changes in a cohort across time
Each study examines different people in the same cohort
Cohort: A group of people with shared characteristics
Advantage: Provides more detail than a trend study
Disadvantage: Determining causality is a problem because you are not studying
the same people over time
Example: How the political views of millennials change over time; babies
Panel Examines the same individuals over time
Can indicate causality, but not definitely determine it
Multiple “waves”, but must account for attrition
Advantage: Can provide far more in-depth understanding (can examine the key
determinants of changes)
Disadvantage: Very expensive
Example: Study aimed at examining the relationship between risky sexual
behaviour and perceptions of peer involvement
Survey mode
With interviewer Response Speed Costs Complexity
Face to face Good Slow (Very) high (Very) high
Phone Reasonable Fast High (Very) high
Without interviewer
Paper & pencil Low Slow Low/moderate Low
Email / web Low Very fast Very low Very high
Types of questions
- Filter questions: Route respondents around questions that don’t apply to them
- Example: If this home has an Internet connection, please continue with
question 6. If it has no Internet connection, please go to question 18.
- Easier with online surveys
- Dichotomous: Respondents must selection one of two options (e.g. yes / no)
- More simplified but life is not that simple
- Open-ended: Allows the respondent to answer however they want
- Get more insight but much harder to code
- Multiple choice: Choose from a few mutually exclusive options
- “Select one” vs. “select many” depends on the necessary information
- Rank order: Ranking questions or statements in order of importance
- Likert scale: Indicate the level of agreement or disagreement
- Semantic-differential scale: Position on a scale between two bipolar adjectives
,Survey wording
- Leading questions: Force the respondent into an assumption that may not be true
- E.g. How stupid is Donald Trump when it comes to immigration related
issues?
- Double-barrelled questions: Asking two questions simultaneously, but allows for
only one answer
- E.g. I felt welcomed by the lecturers and other staff on my first day at UvA
(totally disagree – totally agree)
- Framing questions: When researchers (un)willingly influence the respondent’s
answers
- Negative wording: Easily misunderstood
- E.g. a statistics course should not be a statistics requirement
- Double-negative wording: Negative wording in combination with a double-barrelled
question
- E.g.What is your stance on not allowing selling alcohol?
- Funnel design: Start with broad questions and then go specific (better with phone)
- Inverted funnel design: Start with specific question and then go broad (better
online)
Social desirability bias: A type of response bias that survey respondents answer questions
in a manner that they believe will be viewed favorably by others. The tendency poses a
serious problem with conducting research with self-reports.
To avoid this:
- Indirect questioning
- Show it is ok to answer in a way that it is not desirable (make undesirable answer
less threatening)
Research validity
- Internal validity: No other explanations for causality
- External validity: U nits of analysis are representative
Pre-testing/ pilot testing
Testing the survey first in order to filter out mistakes.
Look at:
- Survey wording
- Aesthetics and design of questionnaire
- Logic and flow of wording
- Words – and their specific meaning(s)
- Length/time it takes to complete questionnaire
How to pre-test?
1. Ask respondents to fill in the questionnaire and add evaluation questions (ask for
feedback)
2. Cognitive interviewing: use think-aloud method and ask respondents what they
think when filling in the questionnaire
3. Ask your supervisor, fellow student or colleague for feedback
, Sampling
A statistical method of obtaining representative data or observations from a population.
- Population: Every person in the group that the researcher wants to study
- Large = difficult
- Sample: Part of a population selected for study
- Hopefully representative
- Census: Study of entire population
- Parameters: Describe a population
- Statistics: Describes a sample
- Sampling frame: List from which a sample is drawn, all the units in a population
- Sampling error: Degree to which the sample statistics differ from the parameters of
the population
Probability sampling
Hopefully representative and generalizable. All units of analysis have an equal chance of
selection, and there is no need for huge samples.
Simple random Removes the researcher as an agent of selection, limiting bias and
sampling allowing for greater representation.
Stratified random Involves subdividing the market into segments (known as strata)
sampling that share homogeneous or very similar characteristics.
A number of respondents from each stratum that is proportional
to the population is then randomly selected for the sample.
Systematic sampling A random starting point followed by selecting every n
th person.
Multistage cluster The taking of samples in stages using smaller and smaller
sampling sampling units at each stage. It is a complex form of cluster
sampling.
Cluster sampling The researcher divides the population into separate groups, called
clusters. Then, a simple random sample of clusters is selected
from the population. Every member from the chosen clusters are
sampled (but not all clusters are used). Clusters can be school
classes, geographical locations etc.
