PROBLEM 7
AN AGENDA FOR PURELY CONFIRMATORY RESEARCH
• cannot find your starting hypothesis in your final results. It makes things all wonky
• Confirmation bias: people seek confirmation rather than disconfirmation of their beliefs
- Ambiguous info is interpreted to be consistent with one’s prior beliefs
- People tend to search for info that confirms rather than disconfirms their hypothesis
- More easily remember info that supports their position
• Hindsight bias: tendency to judge an event as more predictable after it has occurred.
• Indication that bias influences the research process:
- Researchers seek to confirm not falsify their main hypothesis
• Impact of Bias: puts a premium on output quantity
- Researchers attracted to methods and procedures that maximizes the probability of
publication
- Ecologically rational behavior à maximizes the proximal goals of the researcher
• Effects on the field: (concerns)
- Many published results may simply be false à obtained partially by dubious or
inappropriate methods of observation, analysis and reporting
• Main “fairy-tale factor” = the fact that researchers do not commit themselves to a plan of
analysis before they see the data
- Can fine tune the analyses to the data (makes data appear to be more compelling than
really is)
- Increases the probability that presented finding if fictional and non-replicable
• Radical Remedy = preregistration
- To ensure scientific integrity and inoculate the research process against biases of
human reasoning
BAD SCIENCE: Exploratory findings, confirmatory conclusions
• Bad = flawed design, faulty logic,
• Factor the reduces confidence and enthusiasm for scientific finding
- Fact that almost no psychological research is conducted in purely confirmatory
fashion
- Rarely indicate specific analyses they intended to carry out before data collection
, - Explore various transformations of the data, rely on one-sided p values and construct
post-hoc hypothesis that have been tailored to fit observed data
- Cherry picking – they can measure many variables (gender, personality
characteristics, age etc.) and only report those that yield the desired result
- They can include in their papers only those experiments that produce the desired
outcome, even though the experiment was designed as a pilot experiment that could
easily be discarded has the results turned out less favourably
• Some researchers believe they are giving valuable advice
- Researchers want to conform their hypothesis
• For purpose of hypothesis testing, data may only be used once
- When looking for patterns, have used the data to help formulate a specific hypothesis
- No longer appropriate to test hypothesis is helped suggest
- Need fresh data set
• Common areas for mistakes à research question, amount of data collected, prior beliefs of
the researcher
• The interpretation of common statistical tests in terms of type 1 and type 2 error rates is valid
only if:
1. the data were used only once
2. the statistical test was not chosen on the basis of suggestive patterns in the data
- double dipping - if you carry out a hypothesis test on the very data that inspired the
test in the first place - then the statistics are invalid
Exploratory Research
Far left à researchers finding hypothesis in the data by post-hoc theorizing and corresponding
statistics are “wonky”. Overestimating the evidence of the hypothesis
AN AGENDA FOR PURELY CONFIRMATORY RESEARCH
• cannot find your starting hypothesis in your final results. It makes things all wonky
• Confirmation bias: people seek confirmation rather than disconfirmation of their beliefs
- Ambiguous info is interpreted to be consistent with one’s prior beliefs
- People tend to search for info that confirms rather than disconfirms their hypothesis
- More easily remember info that supports their position
• Hindsight bias: tendency to judge an event as more predictable after it has occurred.
• Indication that bias influences the research process:
- Researchers seek to confirm not falsify their main hypothesis
• Impact of Bias: puts a premium on output quantity
- Researchers attracted to methods and procedures that maximizes the probability of
publication
- Ecologically rational behavior à maximizes the proximal goals of the researcher
• Effects on the field: (concerns)
- Many published results may simply be false à obtained partially by dubious or
inappropriate methods of observation, analysis and reporting
• Main “fairy-tale factor” = the fact that researchers do not commit themselves to a plan of
analysis before they see the data
- Can fine tune the analyses to the data (makes data appear to be more compelling than
really is)
- Increases the probability that presented finding if fictional and non-replicable
• Radical Remedy = preregistration
- To ensure scientific integrity and inoculate the research process against biases of
human reasoning
BAD SCIENCE: Exploratory findings, confirmatory conclusions
• Bad = flawed design, faulty logic,
• Factor the reduces confidence and enthusiasm for scientific finding
- Fact that almost no psychological research is conducted in purely confirmatory
fashion
- Rarely indicate specific analyses they intended to carry out before data collection
, - Explore various transformations of the data, rely on one-sided p values and construct
post-hoc hypothesis that have been tailored to fit observed data
- Cherry picking – they can measure many variables (gender, personality
characteristics, age etc.) and only report those that yield the desired result
- They can include in their papers only those experiments that produce the desired
outcome, even though the experiment was designed as a pilot experiment that could
easily be discarded has the results turned out less favourably
• Some researchers believe they are giving valuable advice
- Researchers want to conform their hypothesis
• For purpose of hypothesis testing, data may only be used once
- When looking for patterns, have used the data to help formulate a specific hypothesis
- No longer appropriate to test hypothesis is helped suggest
- Need fresh data set
• Common areas for mistakes à research question, amount of data collected, prior beliefs of
the researcher
• The interpretation of common statistical tests in terms of type 1 and type 2 error rates is valid
only if:
1. the data were used only once
2. the statistical test was not chosen on the basis of suggestive patterns in the data
- double dipping - if you carry out a hypothesis test on the very data that inspired the
test in the first place - then the statistics are invalid
Exploratory Research
Far left à researchers finding hypothesis in the data by post-hoc theorizing and corresponding
statistics are “wonky”. Overestimating the evidence of the hypothesis
