Problem 7 2.3
An agenda for purely confirmatory research
Confirmation bias operates in at least 3 ways”
1. Ambiguous info is readily interpreted to be consistent with one’s prior beliefs
2. People tend to search for info that confirms rather than disconfirms their
preferred hypothesis
3. People more easily remember info that supports their position
Hindsight bias – the tendency to judge an event as more predictable after it has
occurred
It would thus be naïve to believe that, without protective measures, the scientific
research process is somehow exempt from the systematic imperfections of the
human mind
- Researchers seek to confirm, not falsify, their main hypothesis
- When academic survival depends on how many papers one publishes.
Researchers are attracted to methods and procedures that maximise the
probability of publication
◊ Concern – many published results may simply be false, as they have been obtained
partly by dubious or inappropriate methods of observation, analysis and reporting
Fairy tale factor: researchers don’t commit themselves to a plan of analysis before
they see the data fine tuning of data
- Increased probability of fictional findings and reduced replicability
Bad Science: Exploratory Findings, Confirmatory Conclusions
Almost no psychological research is conducted in a purely confirmatory fashion
- Very rarely do psychologist indicate, in advance of data collection, the
specific analysis they intend to carry out
- Cherry picking – they can measure many variables (gender, personality
characteristics, age etc.) and only report those that yield the desired result
- They can include in their papers only those experiments that produce the
desired outcome, even though the experiment was designed as a pilot
experiment that could easily be discarded had the results turned out less
favourably
Exploring various data transformations: using one-sided p values and post hoc
hypotheses to fit the data (easier with software)
- Torture data until they confess, not necessarily bad but it should be clearly
acknowledged
- Statistical law that, for the purpose of hypothesis testing, the data may only
be used one time – can’t explore and then hypothesize on the same data
A hypothesis that is developed on the basis of exploration of the data set is unlikely
to be disproven by the same data
- Thus, one always needs a fresh data set for testing one’s hypothesis
The interpretation of common statistical tests in terms of type 1 and type 2 error
rates is valid only if:
1. the data were used only once
2. the statistical test was not chosen on the basis of suggestive patterns in the data
, - double dipping - if you carry out a hypothesis test on the very data that
inspired the test in the first place - then the statistics are invalid
psychological studies can be placed on a continuum from purely exploratory, where
the hypothesis is found in the data, to purely confirmatory, where the entire analysis
plan has been explicated before the first participant is tested
- initial outcome - poor initial results may encourage exploration
- amount of data collected – more dependent variables encourage more
exploration
- a priori beliefs of the researcher – strong belief in the presence of an effect
encourages exploration when the initial result is ambiguous
Good Science: Confirmatory Conclusions Require Preregistration
researcher should be honest
- researchers may be convinced that they are honest, but they may be blinded
by confirmation and hindsight bias
^ hard to see, thus focus on a more concrete objective: separating exploratory
experiments from confirmatory experiments
By preregistering the study design and the analysis plan, psychology’s main fairy tale
factor (i.e., presenting and analysing exploratory results as if they were
confirmatory) is eliminated in its entirety.
Stage 1 of a research program – researchers should feel free to conduct exploratory
studies and do whatever they please however:
- Exploratory studies cannot be presented as strong evidence in favour of a
particular claim. The focus of exploratory work should be on:
1. describing interesting aspects if the data
2. Determining which tentative findings are of particular interest
3. Proposing efficient ways in which future studies may confirm or
disconfirm the initial exploratory results
Stage 2 of a research program – a purely confirmatory approach is desired
- Before a single ppt is tested, the researcher submits to the online repository
a document that details what dependent variables will be collected and how
the data will be analysed
- When the p values are used, the researcher also needs to indicate exactly
how many ppt will be tested
- When researchers wish to claim that their studies are confirmatory, the
online document then becomes part of the review process
An agenda for purely confirmatory research
Confirmation bias operates in at least 3 ways”
1. Ambiguous info is readily interpreted to be consistent with one’s prior beliefs
2. People tend to search for info that confirms rather than disconfirms their
preferred hypothesis
3. People more easily remember info that supports their position
Hindsight bias – the tendency to judge an event as more predictable after it has
occurred
It would thus be naïve to believe that, without protective measures, the scientific
research process is somehow exempt from the systematic imperfections of the
human mind
- Researchers seek to confirm, not falsify, their main hypothesis
- When academic survival depends on how many papers one publishes.
Researchers are attracted to methods and procedures that maximise the
probability of publication
◊ Concern – many published results may simply be false, as they have been obtained
partly by dubious or inappropriate methods of observation, analysis and reporting
Fairy tale factor: researchers don’t commit themselves to a plan of analysis before
they see the data fine tuning of data
- Increased probability of fictional findings and reduced replicability
Bad Science: Exploratory Findings, Confirmatory Conclusions
Almost no psychological research is conducted in a purely confirmatory fashion
- Very rarely do psychologist indicate, in advance of data collection, the
specific analysis they intend to carry out
- Cherry picking – they can measure many variables (gender, personality
characteristics, age etc.) and only report those that yield the desired result
- They can include in their papers only those experiments that produce the
desired outcome, even though the experiment was designed as a pilot
experiment that could easily be discarded had the results turned out less
favourably
Exploring various data transformations: using one-sided p values and post hoc
hypotheses to fit the data (easier with software)
- Torture data until they confess, not necessarily bad but it should be clearly
acknowledged
- Statistical law that, for the purpose of hypothesis testing, the data may only
be used one time – can’t explore and then hypothesize on the same data
A hypothesis that is developed on the basis of exploration of the data set is unlikely
to be disproven by the same data
- Thus, one always needs a fresh data set for testing one’s hypothesis
The interpretation of common statistical tests in terms of type 1 and type 2 error
rates is valid only if:
1. the data were used only once
2. the statistical test was not chosen on the basis of suggestive patterns in the data
, - double dipping - if you carry out a hypothesis test on the very data that
inspired the test in the first place - then the statistics are invalid
psychological studies can be placed on a continuum from purely exploratory, where
the hypothesis is found in the data, to purely confirmatory, where the entire analysis
plan has been explicated before the first participant is tested
- initial outcome - poor initial results may encourage exploration
- amount of data collected – more dependent variables encourage more
exploration
- a priori beliefs of the researcher – strong belief in the presence of an effect
encourages exploration when the initial result is ambiguous
Good Science: Confirmatory Conclusions Require Preregistration
researcher should be honest
- researchers may be convinced that they are honest, but they may be blinded
by confirmation and hindsight bias
^ hard to see, thus focus on a more concrete objective: separating exploratory
experiments from confirmatory experiments
By preregistering the study design and the analysis plan, psychology’s main fairy tale
factor (i.e., presenting and analysing exploratory results as if they were
confirmatory) is eliminated in its entirety.
Stage 1 of a research program – researchers should feel free to conduct exploratory
studies and do whatever they please however:
- Exploratory studies cannot be presented as strong evidence in favour of a
particular claim. The focus of exploratory work should be on:
1. describing interesting aspects if the data
2. Determining which tentative findings are of particular interest
3. Proposing efficient ways in which future studies may confirm or
disconfirm the initial exploratory results
Stage 2 of a research program – a purely confirmatory approach is desired
- Before a single ppt is tested, the researcher submits to the online repository
a document that details what dependent variables will be collected and how
the data will be analysed
- When the p values are used, the researcher also needs to indicate exactly
how many ppt will be tested
- When researchers wish to claim that their studies are confirmatory, the
online document then becomes part of the review process
