Experimental Research - Lectures
Inhoud
Week 1 ......................................................................................................................................... 2
T-Mobile & Dentsu ................................................................................................................ 5
Week 2 ......................................................................................................................................... 5
Lecture ..................................................................................................................................... 5
Anova ................................................................................................................................... 5
Moderation in ANOVA ........................................................................................................... 8
The fun of interactions ..........................................................................................................11
Week 3 ........................................................................................................................................13
Lecture ....................................................................................................................................13
Models in the experiment .....................................................................................................13
Week 4 ........................................................................................................................................15
Week 4 ........................................................................................................................................15
Lecture – Manipulations and mediation .....................................................................................15
Manipulation and confound checks .......................................................................................15
Controlling for variables ........................................................................................................17
Mediation ............................................................................................................................18
Week 5 – Within and Quasi-experiments .......................................................................................23
Lecture ....................................................................................................................................23
Within-subjects designs ........................................................................................................23
Quasi-Experiments ...............................................................................................................28
Intent to Treat effects (why Google cheats in presenting their effectiveness) ...........................29
Lecture – Causality & design......................................................................................................31
Week 6 ........................................................................................................................................34
Lecture ....................................................................................................................................34
Power ..................................................................................................................................34
Lecture – Problems in research .................................................................................................35
Week 8 ........................................................................................................................................39
Lecture – manipulating people ..................................................................................................39
,WEEK 1
Experimental research has 3 phases:
- 1: Problem statement, hypotheses about the relation between IV and DV
- 2: Design of the experiment: how to manipulate IV’s, control for confounding variables
o How IV and DV are operationalized in experiments
o How to control for confounding variables
o Differences between ‘real’ experiments and quasi-experiments
- 3: data analysis and interpretation of experimental findings
o How to statistically analyze experimental designs using ANOVA
o Interpret results from an experiment
o How the results lead to derive new hypotheses to be tested in a follow-up
experiment
Importance of experiments: why study experiments?
- It’s the best way to get to causal knowledge: main reason
o Economic modelling techniques try to control for variables and get as close as
possible to causality, but in the end only true randomization can do this
▪ Causality: we’re sure that A is what causes B
o Experiments guide investment choices (more accurate effect size estimates than
modelling approaches)
o Big data is past behavior: you make inferences on what you think happened
▪ An experiment is forward looking: you test 2 ads, and next week you’ll find
which one drives sales more
▪ There’s many new ways to change something, but you’ll have to pick 1. Now
we could test multiple versions.
- More and more companies seem to realize point 1
- This helps them learn, grow and prevent mistakes
Goals of science
The goal of science is to find regularities & patterns, to predict outcomes.
We do this by investigating and explaining. A good explanation of why something happens = theory
(documented, supported explanation for observations).
Good theory:
- What: defines constructs
o Examples: temperature & product evaluation example
▪ X (IV) = physical warmth
• Conceptual definition: feeling warm (as compared to feeling cold)
• Operational definition:
o Daily temperature (degrees)
o Holding a warm vs cold object
o Being in a warm vs cold room
▪ Y (DV) = product evaluation
• Conceptual definition: subjective evaluation of a target product
• Operational definition:
o Likelihood to purchase
o Willingness to pay
- How: propositions about the relationship between constructs
o F.e.: physical warmth → positive mood → higher product evaluations
o Because:
▪ Physical warmth creates a better mood
, ▪ In a positive mood we see other things (products) in a more positive light
- Why: arguments that justify the propositions
Why do we research?
- Goal: describing, predicting and explaining behavior
- Types:
o Descriptive: describes behavior, thoughts or feelings
o Correlational: relationships among variables
o Experimental: find whether certain variables cause changes in behavior, thought or
emotions
o Quasi-experimental: close to experimental, but it’s not possible to manipulate the IV
Descriptive:
- Public opinion poll / survey research
- Changes can be measured, if respondents fill in the survey at different points in time
(longitudinal or panel design)
o 30% of males find shipping costs too high, 50% for females
- Most often we want to know if one thing causes another
Causality:
- We want to know whether one thing causes another (not just correlation)
o Will sales increase if we launch this new product?
o Do we sell more when it’s warm / cold outside?
- The most naïve solution is regression (correlation)
Correlational research:
- Investigates the relationship among various (psychological) variables
- Aims to discover correlations between variables
- Used to describe the relationship between 2 or more naturally occurring variables
- But it can’t establish causality
Example: does the weather affect our mood?
- We could measure temperature and mood and their correlations, but there may be other
factors we didn’t measure: maybe humidity drives the effect, because it’s related to
temperature
- Slightly less naïve method: multiple regression (adding control variables)
o But we still miss other factors that may play a role (f.e. food may be better in warmer
Europe, making people happier)
- Another improvement: time series analysis = granger-causality (predictive causality)
o F.e. measure temperature and mood over longer time. If temperature changes are
followed by mood changes (and bigger changes in temperature are followed by
bigger mood changes), we become more sure of the causal relationship
- Does temperature affect consumers’ evaluation of products?
o Higher temperature → more purchases, but it can be affected by the product they’re
selling
- Why care about causality? Does the weather have an effect, or the activities you do then?
o But does it matter?
▪ Not directly to get better at predicting
▪ But it does for causality, and thus truly explaining the effect
o And a better explanation again helps to make better predictions
- 24 months of data (sep 2010 – aug 2012): average temperature per day and intention to
purchase online
, o But is it really temperature driving the effect?
▪ Correlation is present, directionality is clear, but other (confounding)
variables may play a role
3 requirements for causality:
- Correlation is only one of the necessary conditions for causality
- Directionality (logical in time)
- Elimination of extraneous variables (confounding variables): this will always be a problem
Descriptive or correlational?
- Descriptive data is f.e. 40% of site visitors who don’t convert indicate shipping costs are too
high. For amles this was 30%, 50% for females.
- But if correlational data is the relationship between 2 variables, is the gender leading to free
shipping preference not a correlational datapoint (rather than descriptive one)?
o Correlations are descriptives of data
- If correlational data is the relation between 2 variables, they’re just descriptive
Experimental
- Involves manipulating (changing) an IV and assessing potential changes in an outcome
variable (behavior)
- Randomization of subjects to treatment is key: if change occurs, we can infer X causes Y
- Randomization = arbitrarily assigning each participant to one condition of the experiment
o If you assign A to the control, and B to treatment group, the persons aren’t the same
o But if you do this for each subject, the average person in control is the same as the
average person in the treatment group
o With large samples, true randomization creates balance (in age, gender, etc.)
▪ Increasing your sample → difference between groups are cancelled out (on
average they’re the same). Even on things we haven’t talked about
(preferences, hobbies, etc.).
o Any difference we later find, must have occurred because of the treatment
o As the groups are the same on average on all aspects (if sample is large enough), it’s
better than matching techniques economists / marketing modelers tend to use
▪ Because for matching we have to a priori predict possible confounds
(alternative explanations), and match people based on these confounds
▪ But there could be confounds we don’t yet know about
▪ Propensity Score Matching: match people as similar as possible to the other
person, to use as the control person
• Match them on characteristics you think matter, but you still have no
proper randomization, because you don’t know which variables are
possible confounds (just like multiple linear regression), while with
randomization you even control for confounds you don’t know.
Quasi-experimental
- If it’s not possible to manipulate / change the IV
- We try to find situations in the real world where groups are basically the same
o F.e. we can’t test car belts (it’s not ethical to test not using them), so they compared
2 relatively close states, but with different laws on seatbelt use, and then compare
the relation on accidents.
Zwebner, Lee & Goldenberg (2013)
- Temperature manipulation: making people warm / cold by therapeutic pads
- Result: when holding the warm product, the WTP increased compared to a cold one
Inhoud
Week 1 ......................................................................................................................................... 2
T-Mobile & Dentsu ................................................................................................................ 5
Week 2 ......................................................................................................................................... 5
Lecture ..................................................................................................................................... 5
Anova ................................................................................................................................... 5
Moderation in ANOVA ........................................................................................................... 8
The fun of interactions ..........................................................................................................11
Week 3 ........................................................................................................................................13
Lecture ....................................................................................................................................13
Models in the experiment .....................................................................................................13
Week 4 ........................................................................................................................................15
Week 4 ........................................................................................................................................15
Lecture – Manipulations and mediation .....................................................................................15
Manipulation and confound checks .......................................................................................15
Controlling for variables ........................................................................................................17
Mediation ............................................................................................................................18
Week 5 – Within and Quasi-experiments .......................................................................................23
Lecture ....................................................................................................................................23
Within-subjects designs ........................................................................................................23
Quasi-Experiments ...............................................................................................................28
Intent to Treat effects (why Google cheats in presenting their effectiveness) ...........................29
Lecture – Causality & design......................................................................................................31
Week 6 ........................................................................................................................................34
Lecture ....................................................................................................................................34
Power ..................................................................................................................................34
Lecture – Problems in research .................................................................................................35
Week 8 ........................................................................................................................................39
Lecture – manipulating people ..................................................................................................39
,WEEK 1
Experimental research has 3 phases:
- 1: Problem statement, hypotheses about the relation between IV and DV
- 2: Design of the experiment: how to manipulate IV’s, control for confounding variables
o How IV and DV are operationalized in experiments
o How to control for confounding variables
o Differences between ‘real’ experiments and quasi-experiments
- 3: data analysis and interpretation of experimental findings
o How to statistically analyze experimental designs using ANOVA
o Interpret results from an experiment
o How the results lead to derive new hypotheses to be tested in a follow-up
experiment
Importance of experiments: why study experiments?
- It’s the best way to get to causal knowledge: main reason
o Economic modelling techniques try to control for variables and get as close as
possible to causality, but in the end only true randomization can do this
▪ Causality: we’re sure that A is what causes B
o Experiments guide investment choices (more accurate effect size estimates than
modelling approaches)
o Big data is past behavior: you make inferences on what you think happened
▪ An experiment is forward looking: you test 2 ads, and next week you’ll find
which one drives sales more
▪ There’s many new ways to change something, but you’ll have to pick 1. Now
we could test multiple versions.
- More and more companies seem to realize point 1
- This helps them learn, grow and prevent mistakes
Goals of science
The goal of science is to find regularities & patterns, to predict outcomes.
We do this by investigating and explaining. A good explanation of why something happens = theory
(documented, supported explanation for observations).
Good theory:
- What: defines constructs
o Examples: temperature & product evaluation example
▪ X (IV) = physical warmth
• Conceptual definition: feeling warm (as compared to feeling cold)
• Operational definition:
o Daily temperature (degrees)
o Holding a warm vs cold object
o Being in a warm vs cold room
▪ Y (DV) = product evaluation
• Conceptual definition: subjective evaluation of a target product
• Operational definition:
o Likelihood to purchase
o Willingness to pay
- How: propositions about the relationship between constructs
o F.e.: physical warmth → positive mood → higher product evaluations
o Because:
▪ Physical warmth creates a better mood
, ▪ In a positive mood we see other things (products) in a more positive light
- Why: arguments that justify the propositions
Why do we research?
- Goal: describing, predicting and explaining behavior
- Types:
o Descriptive: describes behavior, thoughts or feelings
o Correlational: relationships among variables
o Experimental: find whether certain variables cause changes in behavior, thought or
emotions
o Quasi-experimental: close to experimental, but it’s not possible to manipulate the IV
Descriptive:
- Public opinion poll / survey research
- Changes can be measured, if respondents fill in the survey at different points in time
(longitudinal or panel design)
o 30% of males find shipping costs too high, 50% for females
- Most often we want to know if one thing causes another
Causality:
- We want to know whether one thing causes another (not just correlation)
o Will sales increase if we launch this new product?
o Do we sell more when it’s warm / cold outside?
- The most naïve solution is regression (correlation)
Correlational research:
- Investigates the relationship among various (psychological) variables
- Aims to discover correlations between variables
- Used to describe the relationship between 2 or more naturally occurring variables
- But it can’t establish causality
Example: does the weather affect our mood?
- We could measure temperature and mood and their correlations, but there may be other
factors we didn’t measure: maybe humidity drives the effect, because it’s related to
temperature
- Slightly less naïve method: multiple regression (adding control variables)
o But we still miss other factors that may play a role (f.e. food may be better in warmer
Europe, making people happier)
- Another improvement: time series analysis = granger-causality (predictive causality)
o F.e. measure temperature and mood over longer time. If temperature changes are
followed by mood changes (and bigger changes in temperature are followed by
bigger mood changes), we become more sure of the causal relationship
- Does temperature affect consumers’ evaluation of products?
o Higher temperature → more purchases, but it can be affected by the product they’re
selling
- Why care about causality? Does the weather have an effect, or the activities you do then?
o But does it matter?
▪ Not directly to get better at predicting
▪ But it does for causality, and thus truly explaining the effect
o And a better explanation again helps to make better predictions
- 24 months of data (sep 2010 – aug 2012): average temperature per day and intention to
purchase online
, o But is it really temperature driving the effect?
▪ Correlation is present, directionality is clear, but other (confounding)
variables may play a role
3 requirements for causality:
- Correlation is only one of the necessary conditions for causality
- Directionality (logical in time)
- Elimination of extraneous variables (confounding variables): this will always be a problem
Descriptive or correlational?
- Descriptive data is f.e. 40% of site visitors who don’t convert indicate shipping costs are too
high. For amles this was 30%, 50% for females.
- But if correlational data is the relationship between 2 variables, is the gender leading to free
shipping preference not a correlational datapoint (rather than descriptive one)?
o Correlations are descriptives of data
- If correlational data is the relation between 2 variables, they’re just descriptive
Experimental
- Involves manipulating (changing) an IV and assessing potential changes in an outcome
variable (behavior)
- Randomization of subjects to treatment is key: if change occurs, we can infer X causes Y
- Randomization = arbitrarily assigning each participant to one condition of the experiment
o If you assign A to the control, and B to treatment group, the persons aren’t the same
o But if you do this for each subject, the average person in control is the same as the
average person in the treatment group
o With large samples, true randomization creates balance (in age, gender, etc.)
▪ Increasing your sample → difference between groups are cancelled out (on
average they’re the same). Even on things we haven’t talked about
(preferences, hobbies, etc.).
o Any difference we later find, must have occurred because of the treatment
o As the groups are the same on average on all aspects (if sample is large enough), it’s
better than matching techniques economists / marketing modelers tend to use
▪ Because for matching we have to a priori predict possible confounds
(alternative explanations), and match people based on these confounds
▪ But there could be confounds we don’t yet know about
▪ Propensity Score Matching: match people as similar as possible to the other
person, to use as the control person
• Match them on characteristics you think matter, but you still have no
proper randomization, because you don’t know which variables are
possible confounds (just like multiple linear regression), while with
randomization you even control for confounds you don’t know.
Quasi-experimental
- If it’s not possible to manipulate / change the IV
- We try to find situations in the real world where groups are basically the same
o F.e. we can’t test car belts (it’s not ethical to test not using them), so they compared
2 relatively close states, but with different laws on seatbelt use, and then compare
the relation on accidents.
Zwebner, Lee & Goldenberg (2013)
- Temperature manipulation: making people warm / cold by therapeutic pads
- Result: when holding the warm product, the WTP increased compared to a cold one
