2.3 Problem 7: Psychological Research
De Groot: The meaning of ‘significance’ for different types of research
Exploratory research = get data, then conclude result (e.g., studying medication and seeing
certain patterns, you don’t formulate a hypothesis, you only observe what happens and
conclude from that)
Confirmatory research = plan a hypothesis before data
According to de Groot, there are major differences between exploratory research (ER) &
confirmatory research (CR – hypothesis testing):
Statistical inferences would only be valid for CR. With ER it is allowed to use
statistics, but it has no evidential value
The meaning of the results of statistical tests, applied to psychological experiments, is
a constant source of confusion
In ER, the hypotheses are only formulated afterwards, after research has been carried
out. In CR, this is drawn up in advance
In ER, it is clear which topic is being investigated, but not exactly what will be
investigated. According to de Groot, this is not a question of testing a hypothesis, but
a hypothesis generation
Statistically, it is easiest to test 1 hypothesis that has been prepared in advance.
However, this scenario is rare, because often several hypotheses are tested
simultaneously and because a study contains elements of the material-exploration type
With ER, the goal is to find associations. N is often indefinite, which means that an
exact interpretation of the meaning of positive results (H0 is rejected) is no longer
possible
De Groot makes 3 important points:
1) Analyses of ER invalidate the standard interpretations of hypothesis test results
2) There must be a clear distinction between exploratory & confirmatory research
3) Pre-registration should take place in order to obtain confirmation status as research
Wagenmakers et al.: An agenda for purely confirmatory research
Confirmation bias = people rather look for confirmation than refutation. There are 3 ways in
which confirmation bias can work:
1) Ambiguous info is often interpreted to be consistent w/previous beliefs
2) People tend to look for info that confirms their hypothesis rather than looking for info
that contradicts the hypothesis
3) People remember easy info that supports their position
The presence of a confirmation bias influences science
Publish or perish ( = pressure to publish academic work in order to succeed in an
academic career) also plays a role in this, as the value of a researcher is often
, determined by the number of articles published. So, researchers will use methods &
procedures that increase the likelihood of publication, e.g., by finding significant
results
People also get a hindsight bias: tendency to see an event as more predictable after it
has happened. This problem is that many of the results that have been released can be
simply incorrect when obtained through questionable/inappropriate methods, analysis
and reporting. Replication rates were less than 50% in some studies
Bad Science: Exploratory Findings, Confirmatory Conclusions
Fairy tale factor: not drawing up an analysis plan in advance for viewing the data.
The consequence of this is that researchers can adjust their analyses to the data found,
and as a result, the results appear more convincing than they actually are. This
increases likelihood of presented fictitious findings that are not replicable
A solution for this is pre-registration – this guarantees scientific integrity & protects
research against biases
Cherry picking: measuring a lot of variables (gender, age, etc.), but only look at the
desired variables – choosing only the variables that give a desired result, leaving out
the rest in the report
Massaging of the data: adjusting the data so that a significant difference emerges
(e.g., post-hoc hypotheses). The number of possibilities for this has grown
considerably due to statistical software
Double dipping: using the dataset several times. Once you have used your data to
formulate a hypothesis, this dataset is no longer useful for testing this hypothesis,
because a hypothesis developed from this dataset is unlikely to be refuted by the same
data
So: if you hypothesise the data that inspired the text, your statistics are unreliable
Causes of Bad Science
Exploratory studies & confirmatory studies are used interchangeably
Exploratory can’t serve as strong evidence for theory
Exploratory investigations = actively looking for a relationship, where statistics are
shaky. The hypothesis is extracted from the data
Confirmatory investigations = a healthy form of statistics. The analysis is
determined in advance, then tested with the data. The statistics leave no room for
description
Good Science: Confirmatory Conclusions Require Preregistration
Pre-registration of the research – make an analysis plan before they can see the
data. Then openly post his plan of analysis somewhere, so everyone can see what your
analysis method is
Fairy-tale factor, confirmation bias & hindsight bias are eliminated
Exploratory research isn’t wrong, as long as researchers are honest about their use. It’s
essential for new inventions & progress, it provides inspiration. However, exploratory
research is often reported as confirmatory research, and this is wrong since it can’t be used as
evidence for a hypothesis
De Groot: The meaning of ‘significance’ for different types of research
Exploratory research = get data, then conclude result (e.g., studying medication and seeing
certain patterns, you don’t formulate a hypothesis, you only observe what happens and
conclude from that)
Confirmatory research = plan a hypothesis before data
According to de Groot, there are major differences between exploratory research (ER) &
confirmatory research (CR – hypothesis testing):
Statistical inferences would only be valid for CR. With ER it is allowed to use
statistics, but it has no evidential value
The meaning of the results of statistical tests, applied to psychological experiments, is
a constant source of confusion
In ER, the hypotheses are only formulated afterwards, after research has been carried
out. In CR, this is drawn up in advance
In ER, it is clear which topic is being investigated, but not exactly what will be
investigated. According to de Groot, this is not a question of testing a hypothesis, but
a hypothesis generation
Statistically, it is easiest to test 1 hypothesis that has been prepared in advance.
However, this scenario is rare, because often several hypotheses are tested
simultaneously and because a study contains elements of the material-exploration type
With ER, the goal is to find associations. N is often indefinite, which means that an
exact interpretation of the meaning of positive results (H0 is rejected) is no longer
possible
De Groot makes 3 important points:
1) Analyses of ER invalidate the standard interpretations of hypothesis test results
2) There must be a clear distinction between exploratory & confirmatory research
3) Pre-registration should take place in order to obtain confirmation status as research
Wagenmakers et al.: An agenda for purely confirmatory research
Confirmation bias = people rather look for confirmation than refutation. There are 3 ways in
which confirmation bias can work:
1) Ambiguous info is often interpreted to be consistent w/previous beliefs
2) People tend to look for info that confirms their hypothesis rather than looking for info
that contradicts the hypothesis
3) People remember easy info that supports their position
The presence of a confirmation bias influences science
Publish or perish ( = pressure to publish academic work in order to succeed in an
academic career) also plays a role in this, as the value of a researcher is often
, determined by the number of articles published. So, researchers will use methods &
procedures that increase the likelihood of publication, e.g., by finding significant
results
People also get a hindsight bias: tendency to see an event as more predictable after it
has happened. This problem is that many of the results that have been released can be
simply incorrect when obtained through questionable/inappropriate methods, analysis
and reporting. Replication rates were less than 50% in some studies
Bad Science: Exploratory Findings, Confirmatory Conclusions
Fairy tale factor: not drawing up an analysis plan in advance for viewing the data.
The consequence of this is that researchers can adjust their analyses to the data found,
and as a result, the results appear more convincing than they actually are. This
increases likelihood of presented fictitious findings that are not replicable
A solution for this is pre-registration – this guarantees scientific integrity & protects
research against biases
Cherry picking: measuring a lot of variables (gender, age, etc.), but only look at the
desired variables – choosing only the variables that give a desired result, leaving out
the rest in the report
Massaging of the data: adjusting the data so that a significant difference emerges
(e.g., post-hoc hypotheses). The number of possibilities for this has grown
considerably due to statistical software
Double dipping: using the dataset several times. Once you have used your data to
formulate a hypothesis, this dataset is no longer useful for testing this hypothesis,
because a hypothesis developed from this dataset is unlikely to be refuted by the same
data
So: if you hypothesise the data that inspired the text, your statistics are unreliable
Causes of Bad Science
Exploratory studies & confirmatory studies are used interchangeably
Exploratory can’t serve as strong evidence for theory
Exploratory investigations = actively looking for a relationship, where statistics are
shaky. The hypothesis is extracted from the data
Confirmatory investigations = a healthy form of statistics. The analysis is
determined in advance, then tested with the data. The statistics leave no room for
description
Good Science: Confirmatory Conclusions Require Preregistration
Pre-registration of the research – make an analysis plan before they can see the
data. Then openly post his plan of analysis somewhere, so everyone can see what your
analysis method is
Fairy-tale factor, confirmation bias & hindsight bias are eliminated
Exploratory research isn’t wrong, as long as researchers are honest about their use. It’s
essential for new inventions & progress, it provides inspiration. However, exploratory
research is often reported as confirmatory research, and this is wrong since it can’t be used as
evidence for a hypothesis
