RESEARCH WORKSHOP – SURVEY COMPLETE LECTURE NOTES
WEEK 1
Fowler (2014): chapter 2, 6 and7
Lecture 1
Valid answers? Proper questions!
GOALS RESEARCH WORKSHOP: SURVEY
You can justify the use of survey methods for research
You get practical experience with the empirical cycle
You get to apply quantitative research methods and techniques related to survey
research
You practice with writing a research plan + report
You can operationalize and measure constructs within the context of a survey study
What is survey research?
-> Produce statistics about target population
Why survey research?
TWO INFERENCES: An inference is an idea
I. The answers given by your participants measure the desired concepts or conclusion that's
II. Your sample reflects the population you aim to study drawn from evidence
TWO TYPES OF ERRORS:
1. Random
COMMON SOURCES OF RANDOM ERROR IN SURVEY RESEARCH:
Ambiguous wording of questions
Deviations from the script by the interviewer
Misunderstanding on the part of the interviewee
, Memory problems on the part of the interviewee
2. Systematic
Social Desirability
GETTING ANSWERS: SOCIAL DESIRABILITY
-> Make undesirable answer less threatening:
Everybody makes mistakes in their lives. What about you? Did you commit any serious
crimes in the past that vou may or may not regret?
Have you ever made any of these transgressions? (check all that apply):
Wolf-whistling
Ignoring a homeless person begging for attention
Hitting a person
Stealing
Murder
-> Social desirability varies between groups
How much money do you earn?
How often do you get completely drunk?
How many sexual partners have you had?
"The quality of data will be no better than the most error-prone feature of the survey design"
(Fowler, 2014, p.6)
SURVEY METHODOLOGY IN PRACTICE:
1. Research objective
the aim of the study is to investigate to what extent demographic characteristics are
associated to how people describe the taste of cilantro
2. List of variables
Taste of cilantro
Demographic characteristics (age [ratio], sex [nominal], ancestry[nominal] )
3. Analysis plan
DEVELOPING AND EVALUATING QUESTIONNAIRES
four phases by Fowler (2004):
I. Preliminary question design steps
II. Presurvey evaluation
, III. Design, format, and layout
IV. Field pre-test
EXAMPLE:
I. Preliminary question design steps:
1. Focus groups (e.g., terminology: cilantro or coriander?)
2. Drafting questions
COMMON MISTAKES
Ambiguous terms: 'often', 'regularly', frequently
Long questions
Double-barrelled questions: may be different answers to each part
Very general questions: because they lack a frame of reference
Leading questions: hinting at a preferred response
Negative terms: not', 'never' - especially double negatives
Technical terms: (jargon and acronyms)
OPEN ENDED QUESTIONS
No/less bias from response options
Increased validity
Tiresome for a respondent
Explorative questions
Unanticipated responses
Very important to clarify the question
Length in centimeters
Age in years
Transformations after data collection/ lot of different responses
Sometimes answer categories are preferred
Less detailed, but less tiresome
THE FALLIBLE BRAIN
How often in the last two months did you visit a bar?
How often in the last two months did you take an exam?
How often in the last two months did you go online to find health information?
Over - and under-reporting
Media use: often over-reported
Salient behaviours more accurately reported
INCLUDING "DON'T KNOW" AS A RESPONSE OPTION
-> If you include it, people use it
If you need people to choose one or the other: do not include don't know
If it is possible that people lack the knowledge required to answer a particular question:
including don't know can be sensible
Is there an alternative solution?
Routing - filter questions
o Some questions are not relevant to every interviewee
WEEK 1
Fowler (2014): chapter 2, 6 and7
Lecture 1
Valid answers? Proper questions!
GOALS RESEARCH WORKSHOP: SURVEY
You can justify the use of survey methods for research
You get practical experience with the empirical cycle
You get to apply quantitative research methods and techniques related to survey
research
You practice with writing a research plan + report
You can operationalize and measure constructs within the context of a survey study
What is survey research?
-> Produce statistics about target population
Why survey research?
TWO INFERENCES: An inference is an idea
I. The answers given by your participants measure the desired concepts or conclusion that's
II. Your sample reflects the population you aim to study drawn from evidence
TWO TYPES OF ERRORS:
1. Random
COMMON SOURCES OF RANDOM ERROR IN SURVEY RESEARCH:
Ambiguous wording of questions
Deviations from the script by the interviewer
Misunderstanding on the part of the interviewee
, Memory problems on the part of the interviewee
2. Systematic
Social Desirability
GETTING ANSWERS: SOCIAL DESIRABILITY
-> Make undesirable answer less threatening:
Everybody makes mistakes in their lives. What about you? Did you commit any serious
crimes in the past that vou may or may not regret?
Have you ever made any of these transgressions? (check all that apply):
Wolf-whistling
Ignoring a homeless person begging for attention
Hitting a person
Stealing
Murder
-> Social desirability varies between groups
How much money do you earn?
How often do you get completely drunk?
How many sexual partners have you had?
"The quality of data will be no better than the most error-prone feature of the survey design"
(Fowler, 2014, p.6)
SURVEY METHODOLOGY IN PRACTICE:
1. Research objective
the aim of the study is to investigate to what extent demographic characteristics are
associated to how people describe the taste of cilantro
2. List of variables
Taste of cilantro
Demographic characteristics (age [ratio], sex [nominal], ancestry[nominal] )
3. Analysis plan
DEVELOPING AND EVALUATING QUESTIONNAIRES
four phases by Fowler (2004):
I. Preliminary question design steps
II. Presurvey evaluation
, III. Design, format, and layout
IV. Field pre-test
EXAMPLE:
I. Preliminary question design steps:
1. Focus groups (e.g., terminology: cilantro or coriander?)
2. Drafting questions
COMMON MISTAKES
Ambiguous terms: 'often', 'regularly', frequently
Long questions
Double-barrelled questions: may be different answers to each part
Very general questions: because they lack a frame of reference
Leading questions: hinting at a preferred response
Negative terms: not', 'never' - especially double negatives
Technical terms: (jargon and acronyms)
OPEN ENDED QUESTIONS
No/less bias from response options
Increased validity
Tiresome for a respondent
Explorative questions
Unanticipated responses
Very important to clarify the question
Length in centimeters
Age in years
Transformations after data collection/ lot of different responses
Sometimes answer categories are preferred
Less detailed, but less tiresome
THE FALLIBLE BRAIN
How often in the last two months did you visit a bar?
How often in the last two months did you take an exam?
How often in the last two months did you go online to find health information?
Over - and under-reporting
Media use: often over-reported
Salient behaviours more accurately reported
INCLUDING "DON'T KNOW" AS A RESPONSE OPTION
-> If you include it, people use it
If you need people to choose one or the other: do not include don't know
If it is possible that people lack the knowledge required to answer a particular question:
including don't know can be sensible
Is there an alternative solution?
Routing - filter questions
o Some questions are not relevant to every interviewee
