Onderzoekspracticum 1: Survey (UvA) samenvatting (Cijfer: 7.5)

Onderzoekspracticum: Survey
Januari 2021
Valid answers? Proper questions! (week 1)

Hoorcollege aantekeningen
What is survey research about?
Goal: draw conclusion about population based on a sample | produce statistics/ draw
inferences about a target population | provide information about a population




Bijschrijven

Two inferences: to get information about population
(1) If we pose proper questions, then we should
also assume that the answers given by the
participants measure the desired concept
 Variabele meten door middel van het stellen
van goede vragen die die variabele meten

(2) if we draw a proper sample, then we should
assume that our sample reflects the actual
population
 Goede sample: aannemen dat onze sample
de populatie goed reflecteert

Op deze manier zullen we de kenmerken
achterhalen van de algehele populatie
Together: Characteristics of population = information about the population

We don’t care about individual answers/ respondents, we can use these answers on
a group level to say something about the population as a whole

Common sources of error in surveys:
 A poorly worded question: respondents do not know how to answer -->
gevolg: invalid findings
 Deviations from the script by the interviewer: Face-to-face/ phone surveys
– interview is asking the questions (if script is different everytime, this will lead
to different findings)
 Misunderstanding on the part of the interviewee: if you have unclear
questions (door vage begrippen) --> oplossing: wees duidelijk

, Onderzoekspracticum: Survey
Januari 2021
 Memory problems on the part of the interviewee: mostly with very long
questions (respondent have to think what is being told) --> oplossing: niet
doen
 The way the information is recorded by the interviewer: important source
for error (verjaardagen)
 The way the information is processed, either when answers are coded or
when data are entered into the computer (source of error) --> wees secuur

Two types of error
(1) Random error: the difference due to chance between the observed and the true
value of something – effects our findings in unpredictable ways
(2) Systematic error or bias: consistent difference between observed values and
true values – effects our findings in predictable ways

Both can be applied to sample and to the question level

What errors can be made?
Random Biased (systematic)
Sample: Sampling error Sample bias
 Attitude t.o.v. CW  Attitude t.o.v. CW
Het kan zijn dat de sample een beetje Only Dutch students? Zijn ze allemaal
verschilt van de echte populatie representatief voor de student CW?
(bv: meer vrouwen dan mannen als in Misschien zijn international studenten
de echte populatie) wel heel anders.
-- niet bekend wat de invloed is op de
uiteindelijke resultaten
 It happens


Questions: invalidity Question bias
Vraag: hoeveel liedje geluisterd deze Did you read the literature?
week op spotify? In class: overrepresented?
Vaak overestamted / underestameted
door individuen (lastige vraag) Systematische fout – over het algemeen
-- Individueel niveau: antwoorden zijn overrepresented
niet valide (kloppen vaak niet
helemaal)
-- Groepsniveau: gemiddelde klopt vaak
 Honest mistake


What is wrong with these questions?
Practise!
- Vermijd EN (= double-barreled question) --> oplossing: vraag in 2 vragen splitten
- Vermijd hinten naar het geliefde antwoord (= leading question = biased)
- Vermijd sociaal disaribility (= biased) --> oplossing: extra aandacht aan besteden

, Onderzoekspracticum: Survey
Januari 2021
Question wording
Desigining questions: general rules
 Remember your research question: know what concept to operationalize
 Decide exactly what you want to find out: what relations are you
investigating and how do you want to analyze this (think beforehand)
 Consider your audience: do they understand your questions – pretest!
 Imagine yourself as a respondent: is it clear how to respond to certain
questions – are there questions misleading, biased?
o Would you answer the questions?
o How would you answer the questions?
o Identify any vague or misleading questions

Common mistakes
 Ambiguous terms: subjective terms: often, regularly, frequently
 Long questions: relying on memory --> oplossing: chop it in pieces
 Double-barreled questions: there might be different answers to each part of
the questions that is included in the question
 Very general questions: lack a frame of reference
 Leading questions: hinting at a preferred response already (= expectation
bias)
 Negative terms: not, never, double negatives (= too complicated)
 Technical terms: should be avoided
o Voorbeeld: is your systolic pressure normal at this moment --> moet
zijn: is je bloeddruk normal op dit moment?

A question
 Do respondents have the knowledge to answer the question?
 If you just want a yes/no answer; have you given more possibilities
 Have you thought whether you should include ‘don’t know’ options
 Are you relying too much on the respondent’s memory?

The don’t know option
If we include the don’t know option --> gevolg: this can effect the distribution of all our
responses

Problem? If you use the don’t know option, people are going to use it – do you don’t
know if the people actuallty don’t know, or they are just too lazy – have to keep in
mind

Conclusion: if you use it, people will use it (even though the question might be really
easy)
 Saying ‘’I don’t know’’ varies over e.g., level of education
o highly educated people probably have a higher chance of
understanding the question --> Ev. gevolg: je meet alleen de
hoogopgeleidde omdat je de laagopgeleidde de don’t know option
aanvinken = bias
-- vooral gebruiken voor bijvoorbeeld moeilijke woorden! Alleen dus als de
mogelijkheid er is dat mensen de vraag misschien niet snappen
-- vooral gebruiken voor moeilijke onderwerpen
Midpoints

, Onderzoekspracticum: Survey
Januari 2021
‘’ no opinion ‘’
Similair to the don’t know option

Should u use midpoint or don’t know option?
 If you include don’t know option it, people will use it: (= you can not tell of
people are just lazy or really don’t know)
o If you need people to choose one or the other: do not include don’t
know
o If it is possible that people lack the knowledge required to answer a
particular question: including don’t know can be sensible (moeilijke
begrippen)
o Some will spend less time thinking about difficult questions
 Same holds for middle category
o ‘’neutral’’, ‘’agree nor disagree’’
 Beware:
o Leaving out DK/middle option may cause irritation (especially when its
viable)
o And sometimes middle ground/ don’t know is a possibility

Answers options and measurement levels
 Do you agree or disagree?
o Totally agree – agree – disagree – totally disagree
 Ordinal
 Combine multiple items (at least 3)  interval
 Agree - -- --- --- --- ------ disagree (more options, more then 7 =
o Ordinal  treat as interval
 Why? It is usally quite difficult to collect interval data in social
science
 You can always go from a higher level to a lower level (Age -->
age groups (ordinal) – but you can not go from a lower level to a
higher level) – exception: when we combine multiple items into 1
new scale
 How often do you go to the bar?
o Never – now and then – regularly – often (my often can be different
then your often  however it is easier for the respondent to answer) --
just select the word that fits FOR THEM
o … times per week (= more objective, but harder to answer because
you are relying on memory) – interpretation = easy, it is not subjective
you just compare numbers with each other

Being specific
 How often do you go the bar?
 How detailed should the answer be?
 More precise is more burdensome (and exhausting) – don’t include too many
specific
 Very precise
o How many children? Not hard
o How many songs streamed this week? Too hard
o Rounding off answers