Experimental Research – papers
Inhoud
Week 1 ......................................................................................................................................... 2
Paper - Thomke & Manzi (2014) | The discipline of business experimentation ............................... 2
Paper - Kohavi & Thomke (2017) | The surprising power of online experiments ............................. 3
Week 2 ......................................................................................................................................... 4
Paper – Spiller et. al. (2013) | SPotlights, floodlights, and the magic number zero: simple effects
tests in moderated regression .................................................................................................... 4
Paper – Matz et. al.(2017) | Targeting ......................................................................................... 9
Eckles’ reply .........................................................................................................................10
Matz’ reply to Eckles .............................................................................................................11
Week 5 ........................................................................................................................................11
Paper – Charness et. al. | Within experiments ............................................................................11
Paper – Gordon et. al. | Facebook experiments for ad effectiveness ............................................13
Paper – Konings et. al................................................................................................................14
Week 6 ........................................................................................................................................15
Paper – Simmons et. al. | Researcher degrees of freedom ..........................................................15
Paper – Brysbaert | Power rules of thumb .................................................................................16
Week 8 ........................................................................................................................................17
Paper – Mislavsky et. al. | Ethics of experiments.........................................................................17
Paper – Sunstein (2015) | THe ethics of nudging.........................................................................17
,WEEK 1
PAPER - THOMKE & MANZI (2014) | THE DISCIPLINE OF BUSINESS EXPERIMENTATION
We lack data to inform decisions on innovation. Big data only provides information about past
behavior, not how to react to changes. And managers often rely on their experience or intuition, but
truly innovative ideas often go against them. Managers can do a rigorous test, to discover if a new
product will succeed.
Most organizations don’t test, because they’re reluctant to fund experiments and find it difficult to
execute them. Majority of business is conducted through more complex distribution systems like
store networks and fast-food franchises, and not just testing a direct channel like the internet. The
sample sizes are often too small to get valid results.
In an ideal experiment the IV is manipulated to study changes in the DV, holding all other potential
causes constant.
Questions to ask to know if business experimentation is worth the expense and effort:
- Does the experiment have a clear purpose?
o Only do an experiment if they’re the only way to answer specific questions
o First figure out what you want to learn, then choose the best testing approach, and
finally the scope of the experiment.
o Companies often lack discipline to hone their hypothesis (their hypotheses are
weak), making experiments inefficient
o Also think about side effects: positively (selling more of current products) and
negatively (cannibalizing sales from other profitable items).
- Have stakeholders made a commitment to abide by the results?
o Before testing, stakeholders must agree how to proceed once they have the results.
o They must be willing to walk away from a project if it’s not supported by the data
o A process is needed to ensure test results aren’t ignored, even if they’re
contradictory to executives’ assumptions
o If projects make the cut, first develop test designs, and later they’re conducted
- Is the experiment doable?
o The high causable density of the environment (complexity of variables) can make it
hard to determine cause-and-effect relationship
▪ To deal with this, firms should consider if it’s feasible to use a sample large
enough to average out the effects of all variables, except those in the study.
o Environments constantly change, thus outcomes are often uncertain and unknown
o The required sample size depends on the magnitude of the expected effect
▪ The smaller the expected effect, the more observations needed to detect it
from the surrounding noise, with statistical confidence
- How can we ensure reliable results?
o Companies should trade-off reliability, costs and time.
o 3 methods to reduce trade-offs, thus increase reliability of the results:
▪ Randomized field trials:
• Randomly divide people in 2 groups, and give on the treatment
• Prevents systematic bias and evenly spreads remaining, unknown,
potential causes of the outcome between test and control groups.
▪ Blind tests:
• Minimizes biases and increases reliability further
, • Prevents Hawthorne effect: tendency of participants to modify their
behavior, (un)consciously, when they’re aware they’re part of an
experiment
• Double-blind tests: neither experimenters, nor test subjects are
aware of which participants are in the test group, and which are in
the control group.
▪ Big data: utilizing specialized algorithms with multiple sets of big data, can
help by:
• Filtering statistical noise and identify cause-and-effect relationships.
Experiments ideally employ many samples, using big data can help.
• With small samples, we should match test subjects to control
subjects, which depends on identifying many variables to
characterize test subjects, with which big data can help. Then we
build control groups with all elements of the test group, except for
what’s tested, to determine the influencing factors.
• Targeting the best opportunities (highest ROI’s, what investments to
avoid)
• Tailoring the program: characterizing program components that are
more or less effective
o Repeatability is important: others conducting the same test, should get familiar
results.
- Have we gotten the most value out of the experiment?
o It’s not what works, but what works where?
▪ Roll out the program only in stores most similar to test stores with the best
results
▪ Value engineering: implement components with a high ROI
o Just the experiment is not enough, the value comes from analyzing and exploiting
the data
PAPER - KOHAVI & THOMKE (2017) | THE SURPRISING POWER OF ONLINE
EXPERIMENTS
Lessons on how to design and execute A/B tests:
- Appreciate the value of A/B tests
o Using online A/B tests, companies have access to large customer samples (if you
have many website visitors). Allowing to iterate rapidly, fail fast, and pivot.
o Managers need to understand:
▪ Tiny changes can have big impact
• Online success means getting many small changes right, rather than
1 big investment
▪ Experiments can guide investment decisions
• Help decide how much investment in a potential improvement is
optimal
• But can the value of improvement be quantified? (F.e. how many
people should be working on a task)
• They help make trade-offs
o F.e. improve Bing’s results, but they slow the response time
- Build a large-scale capability
o Majority of ideas fail in experiments, even experts (10-20% lead to positive results)
o We need to experiment with everything, to assure changes aren’t degrading nor
have no effects
Inhoud
Week 1 ......................................................................................................................................... 2
Paper - Thomke & Manzi (2014) | The discipline of business experimentation ............................... 2
Paper - Kohavi & Thomke (2017) | The surprising power of online experiments ............................. 3
Week 2 ......................................................................................................................................... 4
Paper – Spiller et. al. (2013) | SPotlights, floodlights, and the magic number zero: simple effects
tests in moderated regression .................................................................................................... 4
Paper – Matz et. al.(2017) | Targeting ......................................................................................... 9
Eckles’ reply .........................................................................................................................10
Matz’ reply to Eckles .............................................................................................................11
Week 5 ........................................................................................................................................11
Paper – Charness et. al. | Within experiments ............................................................................11
Paper – Gordon et. al. | Facebook experiments for ad effectiveness ............................................13
Paper – Konings et. al................................................................................................................14
Week 6 ........................................................................................................................................15
Paper – Simmons et. al. | Researcher degrees of freedom ..........................................................15
Paper – Brysbaert | Power rules of thumb .................................................................................16
Week 8 ........................................................................................................................................17
Paper – Mislavsky et. al. | Ethics of experiments.........................................................................17
Paper – Sunstein (2015) | THe ethics of nudging.........................................................................17
,WEEK 1
PAPER - THOMKE & MANZI (2014) | THE DISCIPLINE OF BUSINESS EXPERIMENTATION
We lack data to inform decisions on innovation. Big data only provides information about past
behavior, not how to react to changes. And managers often rely on their experience or intuition, but
truly innovative ideas often go against them. Managers can do a rigorous test, to discover if a new
product will succeed.
Most organizations don’t test, because they’re reluctant to fund experiments and find it difficult to
execute them. Majority of business is conducted through more complex distribution systems like
store networks and fast-food franchises, and not just testing a direct channel like the internet. The
sample sizes are often too small to get valid results.
In an ideal experiment the IV is manipulated to study changes in the DV, holding all other potential
causes constant.
Questions to ask to know if business experimentation is worth the expense and effort:
- Does the experiment have a clear purpose?
o Only do an experiment if they’re the only way to answer specific questions
o First figure out what you want to learn, then choose the best testing approach, and
finally the scope of the experiment.
o Companies often lack discipline to hone their hypothesis (their hypotheses are
weak), making experiments inefficient
o Also think about side effects: positively (selling more of current products) and
negatively (cannibalizing sales from other profitable items).
- Have stakeholders made a commitment to abide by the results?
o Before testing, stakeholders must agree how to proceed once they have the results.
o They must be willing to walk away from a project if it’s not supported by the data
o A process is needed to ensure test results aren’t ignored, even if they’re
contradictory to executives’ assumptions
o If projects make the cut, first develop test designs, and later they’re conducted
- Is the experiment doable?
o The high causable density of the environment (complexity of variables) can make it
hard to determine cause-and-effect relationship
▪ To deal with this, firms should consider if it’s feasible to use a sample large
enough to average out the effects of all variables, except those in the study.
o Environments constantly change, thus outcomes are often uncertain and unknown
o The required sample size depends on the magnitude of the expected effect
▪ The smaller the expected effect, the more observations needed to detect it
from the surrounding noise, with statistical confidence
- How can we ensure reliable results?
o Companies should trade-off reliability, costs and time.
o 3 methods to reduce trade-offs, thus increase reliability of the results:
▪ Randomized field trials:
• Randomly divide people in 2 groups, and give on the treatment
• Prevents systematic bias and evenly spreads remaining, unknown,
potential causes of the outcome between test and control groups.
▪ Blind tests:
• Minimizes biases and increases reliability further
, • Prevents Hawthorne effect: tendency of participants to modify their
behavior, (un)consciously, when they’re aware they’re part of an
experiment
• Double-blind tests: neither experimenters, nor test subjects are
aware of which participants are in the test group, and which are in
the control group.
▪ Big data: utilizing specialized algorithms with multiple sets of big data, can
help by:
• Filtering statistical noise and identify cause-and-effect relationships.
Experiments ideally employ many samples, using big data can help.
• With small samples, we should match test subjects to control
subjects, which depends on identifying many variables to
characterize test subjects, with which big data can help. Then we
build control groups with all elements of the test group, except for
what’s tested, to determine the influencing factors.
• Targeting the best opportunities (highest ROI’s, what investments to
avoid)
• Tailoring the program: characterizing program components that are
more or less effective
o Repeatability is important: others conducting the same test, should get familiar
results.
- Have we gotten the most value out of the experiment?
o It’s not what works, but what works where?
▪ Roll out the program only in stores most similar to test stores with the best
results
▪ Value engineering: implement components with a high ROI
o Just the experiment is not enough, the value comes from analyzing and exploiting
the data
PAPER - KOHAVI & THOMKE (2017) | THE SURPRISING POWER OF ONLINE
EXPERIMENTS
Lessons on how to design and execute A/B tests:
- Appreciate the value of A/B tests
o Using online A/B tests, companies have access to large customer samples (if you
have many website visitors). Allowing to iterate rapidly, fail fast, and pivot.
o Managers need to understand:
▪ Tiny changes can have big impact
• Online success means getting many small changes right, rather than
1 big investment
▪ Experiments can guide investment decisions
• Help decide how much investment in a potential improvement is
optimal
• But can the value of improvement be quantified? (F.e. how many
people should be working on a task)
• They help make trade-offs
o F.e. improve Bing’s results, but they slow the response time
- Build a large-scale capability
o Majority of ideas fail in experiments, even experts (10-20% lead to positive results)
o We need to experiment with everything, to assure changes aren’t degrading nor
have no effects
