Experimental Research – Tilburg University
Inhoud
Week 1 ......................................................................................................................................... 3
Lecture ..................................................................................................................................... 3
T-Mobile & Dentsu ................................................................................................................ 6
Stat Video – Basics of why we test .............................................................................................. 6
Stat video – T-Test (independent samples) .................................................................................. 7
Paper - Thomke & Manzi (2014) | The discipline of business experimentation .............................. 9
Paper - Kohavi & Thomke (2017) | The surprising power of online experiments ...........................11
Week 2 ........................................................................................................................................12
Lecture ....................................................................................................................................12
Anova ..................................................................................................................................12
Moderation in ANOVA ..........................................................................................................15
The fun of interactions ..........................................................................................................18
Stat Video – ANOVA .................................................................................................................20
Stat Video - MANOVA (Multiple DV’s) ........................................................................................22
Stat Video - Moderation (interaction) in the ANOVA ...................................................................23
Stat video – Moderation with one categorical and one continuous (scale) variable.......................27
Paper – Spiller et. al. (2013) | SPotlights, floodlights, and the magic number zero: simple effects
tests in moderated regression ...................................................................................................27
Paper – Matz et. al.(2017) | Targeting .......................................................................................32
Eckles’ reply .........................................................................................................................33
Matz’ reply to Eckles .............................................................................................................34
Week 3 ........................................................................................................................................34
Lecture ....................................................................................................................................34
Models in the experiment .....................................................................................................34
On measurement video ............................................................................................................36
Week 4 ........................................................................................................................................38
Lecture – Manipulations and mediation .....................................................................................38
Manipulation and confound checks .......................................................................................38
Controlling for variables ........................................................................................................40
Mediation ............................................................................................................................41
Stat video – ANCOVA (ANOVA with a covariate) .........................................................................45
Stat video: Mediation ...............................................................................................................46
Stat video: Dichotomous (binary) DV (in an experiment) .............................................................46
Week 5 – Within and Quasi-experiments .......................................................................................47
,Experimental Research – Tilburg University
Lecture ....................................................................................................................................47
Within-subjects designs ........................................................................................................47
Quasi-Experiments ...............................................................................................................52
Intent to Treat effects (why Google cheats in presenting their effectiveness) ...........................54
Lecture – Causality & design......................................................................................................55
Stat video: Within-subjects t-test ..............................................................................................58
Paper – Charness et. al. | Within experiments ............................................................................58
Paper – Gordon et. al. | Facebook experiments for ad effectiveness ............................................59
Paper – Konings et. al. ..............................................................................................................61
Week 6 ........................................................................................................................................62
Lecture ....................................................................................................................................62
Power ..................................................................................................................................62
Lecture – Problems in research .................................................................................................64
Paper – Simmons et. al. | Researcher degrees of freedom ..........................................................67
Paper – Brysbaert | Power rules of thumb .................................................................................68
Week 8 ........................................................................................................................................69
Lecture – manipulating people ..................................................................................................69
Paper – Mislavsky et. al. | Ethics of experiments ........................................................................71
Paper – Sunstein (2015) | THe ethics of nudging ........................................................................71
,Experimental Research – Tilburg University
Experimental Research - Summary
WEEK 1
LECTURE
Experimental research has 3 phases:
- 1: Problem statement, hypotheses about the relation between IV and DV
- 2: Design of the experiment: how to manipulate IV’s, control for confounding variables
o How IV and DV are operationalized in experiments
o How to control for confounding variables
o Differences between ‘real’ experiments and quasi-experiments
- 3: data analysis and interpretation of experimental findings
o How to statistically analyze experimental designs using ANOVA
o Interpret results from an experiment
o How the results lead to derive new hypotheses to be tested in a follow-up
experiment
Importance of experiments: why study experiments?
- It’s the best way to get to causal knowledge: main reason
o Economic modelling techniques try to control for variables and get as close as
possible to causality, but in the end only true randomization can do this
▪ Causality: we’re sure that A is what causes B
o Experiments guide investment choices (more accurate effect size estimates than
modelling approaches)
o Big data is past behavior: you make inferences on what you think happened
▪ An experiment is forward looking: you test 2 ads, and next week you’ll find
which one drives sales more
▪ There’s many new ways to change something, but you’ll have to pick 1. Now
we could test multiple versions.
- More and more companies seem to realize point 1
- This helps them learn, grow and prevent mistakes
Goals of science
The goal of science is to find regularities & patterns, to predict outcomes.
We do this by investigating and explaining. A good explanation of why something happens = theory
(documented, supported explanation for observations).
Good theory:
- What: defines constructs
o Examples: temperature & product evaluation example
▪ X (IV) = physical warmth
• Conceptual definition: feeling warm (as compared to feeling cold)
• Operational definition:
o Daily temperature (degrees)
o Holding a warm vs cold object
o Being in a warm vs cold room
▪ Y (DV) = product evaluation
• Conceptual definition: subjective evaluation of a target product
• Operational definition:
o Likelihood to purchase
, Experimental Research – Tilburg University
o Willingness to pay
- How: propositions about the relationship between constructs
o F.e.: physical warmth → positive mood → higher product evaluations
o Because:
▪ Physical warmth creates a better mood
▪ In a positive mood we see other things (products) in a more positive light
- Why: arguments that justify the propositions
Why do we research?
- Goal: describing, predicting and explaining behavior
- Types:
o Descriptive: describes behavior, thoughts or feelings
o Correlational: relationships among variables
o Experimental: find whether certain variables cause changes in behavior, thought or
emotions
o Quasi-experimental: close to experimental, but it’s not possible to manipulate the IV
Descriptive:
- Public opinion poll / survey research
- Changes can be measured, if respondents fill in the survey at different points in time
(longitudinal or panel design)
o 30% of males find shipping costs too high, 50% for females
- Most often we want to know if one thing causes another
Causality:
- We want to know whether one thing causes another (not just correlation)
o Will sales increase if we launch this new product?
o Do we sell more when it’s warm / cold outside?
- The most naïve solution is regression (correlation)
Correlational research:
- Investigates the relationship among various (psychological) variables
- Aims to discover correlations between variables
- Used to describe the relationship between 2 or more naturally occurring variables
- But it can’t establish causality
Example: does the weather affect our mood?
- We could measure temperature and mood and their correlations, but there may be other
factors we didn’t measure: maybe humidity drives the effect, because it’s related to
temperature
- Slightly less naïve method: multiple regression (adding control variables)
o But we still miss other factors that may play a role (f.e. food may be better in warmer
Europe, making people happier)
- Another improvement: time series analysis = granger-causality (predictive causality)
o F.e. measure temperature and mood over longer time. If temperature changes are
followed by mood changes (and bigger changes in temperature are followed by
bigger mood changes), we become more sure of the causal relationship
- Does temperature affect consumers’ evaluation of products?
o Higher temperature → more purchases, but it can be affected by the product they’re
selling
- Why care about causality? Does the weather have an effect, or the activities you do then?
o But does it matter?
Inhoud
Week 1 ......................................................................................................................................... 3
Lecture ..................................................................................................................................... 3
T-Mobile & Dentsu ................................................................................................................ 6
Stat Video – Basics of why we test .............................................................................................. 6
Stat video – T-Test (independent samples) .................................................................................. 7
Paper - Thomke & Manzi (2014) | The discipline of business experimentation .............................. 9
Paper - Kohavi & Thomke (2017) | The surprising power of online experiments ...........................11
Week 2 ........................................................................................................................................12
Lecture ....................................................................................................................................12
Anova ..................................................................................................................................12
Moderation in ANOVA ..........................................................................................................15
The fun of interactions ..........................................................................................................18
Stat Video – ANOVA .................................................................................................................20
Stat Video - MANOVA (Multiple DV’s) ........................................................................................22
Stat Video - Moderation (interaction) in the ANOVA ...................................................................23
Stat video – Moderation with one categorical and one continuous (scale) variable.......................27
Paper – Spiller et. al. (2013) | SPotlights, floodlights, and the magic number zero: simple effects
tests in moderated regression ...................................................................................................27
Paper – Matz et. al.(2017) | Targeting .......................................................................................32
Eckles’ reply .........................................................................................................................33
Matz’ reply to Eckles .............................................................................................................34
Week 3 ........................................................................................................................................34
Lecture ....................................................................................................................................34
Models in the experiment .....................................................................................................34
On measurement video ............................................................................................................36
Week 4 ........................................................................................................................................38
Lecture – Manipulations and mediation .....................................................................................38
Manipulation and confound checks .......................................................................................38
Controlling for variables ........................................................................................................40
Mediation ............................................................................................................................41
Stat video – ANCOVA (ANOVA with a covariate) .........................................................................45
Stat video: Mediation ...............................................................................................................46
Stat video: Dichotomous (binary) DV (in an experiment) .............................................................46
Week 5 – Within and Quasi-experiments .......................................................................................47
,Experimental Research – Tilburg University
Lecture ....................................................................................................................................47
Within-subjects designs ........................................................................................................47
Quasi-Experiments ...............................................................................................................52
Intent to Treat effects (why Google cheats in presenting their effectiveness) ...........................54
Lecture – Causality & design......................................................................................................55
Stat video: Within-subjects t-test ..............................................................................................58
Paper – Charness et. al. | Within experiments ............................................................................58
Paper – Gordon et. al. | Facebook experiments for ad effectiveness ............................................59
Paper – Konings et. al. ..............................................................................................................61
Week 6 ........................................................................................................................................62
Lecture ....................................................................................................................................62
Power ..................................................................................................................................62
Lecture – Problems in research .................................................................................................64
Paper – Simmons et. al. | Researcher degrees of freedom ..........................................................67
Paper – Brysbaert | Power rules of thumb .................................................................................68
Week 8 ........................................................................................................................................69
Lecture – manipulating people ..................................................................................................69
Paper – Mislavsky et. al. | Ethics of experiments ........................................................................71
Paper – Sunstein (2015) | THe ethics of nudging ........................................................................71
,Experimental Research – Tilburg University
Experimental Research - Summary
WEEK 1
LECTURE
Experimental research has 3 phases:
- 1: Problem statement, hypotheses about the relation between IV and DV
- 2: Design of the experiment: how to manipulate IV’s, control for confounding variables
o How IV and DV are operationalized in experiments
o How to control for confounding variables
o Differences between ‘real’ experiments and quasi-experiments
- 3: data analysis and interpretation of experimental findings
o How to statistically analyze experimental designs using ANOVA
o Interpret results from an experiment
o How the results lead to derive new hypotheses to be tested in a follow-up
experiment
Importance of experiments: why study experiments?
- It’s the best way to get to causal knowledge: main reason
o Economic modelling techniques try to control for variables and get as close as
possible to causality, but in the end only true randomization can do this
▪ Causality: we’re sure that A is what causes B
o Experiments guide investment choices (more accurate effect size estimates than
modelling approaches)
o Big data is past behavior: you make inferences on what you think happened
▪ An experiment is forward looking: you test 2 ads, and next week you’ll find
which one drives sales more
▪ There’s many new ways to change something, but you’ll have to pick 1. Now
we could test multiple versions.
- More and more companies seem to realize point 1
- This helps them learn, grow and prevent mistakes
Goals of science
The goal of science is to find regularities & patterns, to predict outcomes.
We do this by investigating and explaining. A good explanation of why something happens = theory
(documented, supported explanation for observations).
Good theory:
- What: defines constructs
o Examples: temperature & product evaluation example
▪ X (IV) = physical warmth
• Conceptual definition: feeling warm (as compared to feeling cold)
• Operational definition:
o Daily temperature (degrees)
o Holding a warm vs cold object
o Being in a warm vs cold room
▪ Y (DV) = product evaluation
• Conceptual definition: subjective evaluation of a target product
• Operational definition:
o Likelihood to purchase
, Experimental Research – Tilburg University
o Willingness to pay
- How: propositions about the relationship between constructs
o F.e.: physical warmth → positive mood → higher product evaluations
o Because:
▪ Physical warmth creates a better mood
▪ In a positive mood we see other things (products) in a more positive light
- Why: arguments that justify the propositions
Why do we research?
- Goal: describing, predicting and explaining behavior
- Types:
o Descriptive: describes behavior, thoughts or feelings
o Correlational: relationships among variables
o Experimental: find whether certain variables cause changes in behavior, thought or
emotions
o Quasi-experimental: close to experimental, but it’s not possible to manipulate the IV
Descriptive:
- Public opinion poll / survey research
- Changes can be measured, if respondents fill in the survey at different points in time
(longitudinal or panel design)
o 30% of males find shipping costs too high, 50% for females
- Most often we want to know if one thing causes another
Causality:
- We want to know whether one thing causes another (not just correlation)
o Will sales increase if we launch this new product?
o Do we sell more when it’s warm / cold outside?
- The most naïve solution is regression (correlation)
Correlational research:
- Investigates the relationship among various (psychological) variables
- Aims to discover correlations between variables
- Used to describe the relationship between 2 or more naturally occurring variables
- But it can’t establish causality
Example: does the weather affect our mood?
- We could measure temperature and mood and their correlations, but there may be other
factors we didn’t measure: maybe humidity drives the effect, because it’s related to
temperature
- Slightly less naïve method: multiple regression (adding control variables)
o But we still miss other factors that may play a role (f.e. food may be better in warmer
Europe, making people happier)
- Another improvement: time series analysis = granger-causality (predictive causality)
o F.e. measure temperature and mood over longer time. If temperature changes are
followed by mood changes (and bigger changes in temperature are followed by
bigger mood changes), we become more sure of the causal relationship
- Does temperature affect consumers’ evaluation of products?
o Higher temperature → more purchases, but it can be affected by the product they’re
selling
- Why care about causality? Does the weather have an effect, or the activities you do then?
o But does it matter?
