Problem 7 – only articles
ARTICLE: THE MEANING OF SIGNIFICANCE FOR DIFFERENT TYPES OF RESEARCH (de Groot)
- The style of research for the fourth phase of the empirical cycle is called hypothesis
testing research (confirmatory research)
- This should be distinguished from material exploration: which is more like hypothesis
generation exploratory research
- Statistical inference is only logical for confirmatory research
- One can apply statistical tests to exploratory research too, but should know that they
don’t have evidential impact
ARTICLE: AN AGENDA FOR PURELY CONFIRMATORY RESEARCH (Wagenmakers)
- Confirmation bias: people tend to seek confirmation rather than disconfirmation of
their beliefs
o Ambiguous information is interpreted to be consistent with one’s prior beliefs
o People tend to search for information that confirms their hypothesis
o People more easily remember information that support their position
- Hindsight bias: tendency to judge an event as more predictable after it has occurred
o Hindsight bias often makes exploratory findings appear perfectly sensible
- When academic survival depends on how many papers you publish, researchers are
attracted to methods that will maximize the probability of publication
- “Fairy-tale factor”: psychologists don’t commit themselves to a method of analysis
before they see the actual data
- They fine tune the analysis to the data to obtain the desired result
- The fairy-tale factor increases the probability of false findings that are on-replicable
- Researchers do “cherry picking”: they measure may variables and only report the
ones that yield the desired result
- Fairy-tale factor can be eliminated by preregistering: you publish a protocol for your
study, the methods section of your paper somewhere public before you start
- Researchers explore various transformations of data – rely on one-sided p values and
construct post-hoc test that are tailored to the observed data
- For hypothesis testing data should be only used once – you can’t refute a hypothesis
if you use the same data set that was used for the exploration of that hypothesis
- Interpretations of tests is only valid if the data was used once, and the tests were not
chosen based on suggestive patterns of data
- A continuum:
- From purely exploratory: hypothesis is found in the data -----
---- to purely confirmatory: the entire analysis plan has been clarified before the first
participant is tested
- Exploratory research is not disapproved if it’s openly acknowledged, it’s actually very
useful for new discoveries and scientific progress
- However, they can’t be presented as strong evidence in favor of a claim so future
studies should confirm or disconfirm the initial exploratory results –> a purely
confirmatory research is needed for this
, ARTICLE: PUBLICATION AND OTHER REPORTING BIASES IN COGNITIVE SCIENCES:
DETECTION, PREVALENCE AND PREVENTION (Ioannidis)
- Publication bias and selective reporting bias: refer to differential choice to publish
studies or report particular results, depending on the nature of findings
- There are several forms of biases:
o Study publication bias: studies are less likely to be published when they reach
nonstatistically significant findings
o Selective outcome reporting bias: multiple outcomes are evaluated in a study
and nonsignificant outcomes are less likely to be published than statistically
significant ones
o Selective analysis reporting bias: certain data are analyzed using different
analytical options and publications favors the more impressive and significant
variants of the results
- Biases in psychology:
o In clinical trials, literature is often influenced by financial interests from
manufacturer sponsors
o E.g. although almost half of the trials on antidepressants for depression have
negative results in the FDA records, these results are either unpublished or
published with distorted reporting that shows them positive
- Tests for publication biases:
o Tests of small-study effects:
They evaluate studies included in the same meta-analysis and assess
whether effect sizes are related to study size
When small studies have larger effects, this may reflect in publication
or selective reporting biases
o Selection model:
Evaluates whether pattern of results from several studies suggest an
underlying filtering process, such as results from nonpublished articles
because they were not statistically significant
o Excess significance:
Evaluates whether the number of statistically significant results is
estimated by summing the calculated power of all the considered
studies
- Methods for correction of bias:
o Estimate the extrapolated effect size for a study
o Impute missing nonsignificant studies in meta-analyses to estimate correct
effect size
o Model explicitly the impact of multiple biases on data
- Approaches to prevent bias:
o Noble intentions are not sufficient enough to decrease biases when people
are not able to recognize or control their own biases
ARTICLE: THE MEANING OF SIGNIFICANCE FOR DIFFERENT TYPES OF RESEARCH (de Groot)
- The style of research for the fourth phase of the empirical cycle is called hypothesis
testing research (confirmatory research)
- This should be distinguished from material exploration: which is more like hypothesis
generation exploratory research
- Statistical inference is only logical for confirmatory research
- One can apply statistical tests to exploratory research too, but should know that they
don’t have evidential impact
ARTICLE: AN AGENDA FOR PURELY CONFIRMATORY RESEARCH (Wagenmakers)
- Confirmation bias: people tend to seek confirmation rather than disconfirmation of
their beliefs
o Ambiguous information is interpreted to be consistent with one’s prior beliefs
o People tend to search for information that confirms their hypothesis
o People more easily remember information that support their position
- Hindsight bias: tendency to judge an event as more predictable after it has occurred
o Hindsight bias often makes exploratory findings appear perfectly sensible
- When academic survival depends on how many papers you publish, researchers are
attracted to methods that will maximize the probability of publication
- “Fairy-tale factor”: psychologists don’t commit themselves to a method of analysis
before they see the actual data
- They fine tune the analysis to the data to obtain the desired result
- The fairy-tale factor increases the probability of false findings that are on-replicable
- Researchers do “cherry picking”: they measure may variables and only report the
ones that yield the desired result
- Fairy-tale factor can be eliminated by preregistering: you publish a protocol for your
study, the methods section of your paper somewhere public before you start
- Researchers explore various transformations of data – rely on one-sided p values and
construct post-hoc test that are tailored to the observed data
- For hypothesis testing data should be only used once – you can’t refute a hypothesis
if you use the same data set that was used for the exploration of that hypothesis
- Interpretations of tests is only valid if the data was used once, and the tests were not
chosen based on suggestive patterns of data
- A continuum:
- From purely exploratory: hypothesis is found in the data -----
---- to purely confirmatory: the entire analysis plan has been clarified before the first
participant is tested
- Exploratory research is not disapproved if it’s openly acknowledged, it’s actually very
useful for new discoveries and scientific progress
- However, they can’t be presented as strong evidence in favor of a claim so future
studies should confirm or disconfirm the initial exploratory results –> a purely
confirmatory research is needed for this
, ARTICLE: PUBLICATION AND OTHER REPORTING BIASES IN COGNITIVE SCIENCES:
DETECTION, PREVALENCE AND PREVENTION (Ioannidis)
- Publication bias and selective reporting bias: refer to differential choice to publish
studies or report particular results, depending on the nature of findings
- There are several forms of biases:
o Study publication bias: studies are less likely to be published when they reach
nonstatistically significant findings
o Selective outcome reporting bias: multiple outcomes are evaluated in a study
and nonsignificant outcomes are less likely to be published than statistically
significant ones
o Selective analysis reporting bias: certain data are analyzed using different
analytical options and publications favors the more impressive and significant
variants of the results
- Biases in psychology:
o In clinical trials, literature is often influenced by financial interests from
manufacturer sponsors
o E.g. although almost half of the trials on antidepressants for depression have
negative results in the FDA records, these results are either unpublished or
published with distorted reporting that shows them positive
- Tests for publication biases:
o Tests of small-study effects:
They evaluate studies included in the same meta-analysis and assess
whether effect sizes are related to study size
When small studies have larger effects, this may reflect in publication
or selective reporting biases
o Selection model:
Evaluates whether pattern of results from several studies suggest an
underlying filtering process, such as results from nonpublished articles
because they were not statistically significant
o Excess significance:
Evaluates whether the number of statistically significant results is
estimated by summing the calculated power of all the considered
studies
- Methods for correction of bias:
o Estimate the extrapolated effect size for a study
o Impute missing nonsignificant studies in meta-analyses to estimate correct
effect size
o Model explicitly the impact of multiple biases on data
- Approaches to prevent bias:
o Noble intentions are not sufficient enough to decrease biases when people
are not able to recognize or control their own biases
