Meltzoff (2018) – Seeking Trustworthy Knowledge + Inferences, Conclusions, and
the Research Report
Critical thinking = self-directed, self-disciplined, self-monitored, and self-corrective thinking
- Actively anticipating what one expects to read
- Using explicit rules to judge whether the expectations are met
- Constantly monitoring if one’s expectations are being not
- Having clear and justifiable reasons when one makes an evaluative judgement about the
strengths and limitations of an article
Deciding what is right or true is based on several factors
- Habit or tenacity: a state of certainty is preferred over an unsettled state of mind that
accompanies doubt
- Authority: taking judgment of one or more authorities as sufficient evidence
- Democratic judgment: trusting in the judgment of many (can lead to groupthink)
- Reasoning (deductive or inductive)
- Sensory information: personal experience
- Empirical and experimental methods: combines rationality and observation, and is the best
way to produce knowledge that is unbiased
- Reverence for the written word
Interpreting effect sizes
- Almost any effect size can be made to look large or small depending on the choice o the
contrasting effect sizes
- One must decide whether the contrasts were chosen well
Red flags in research reports
- The abstract should not present a false or overly rosy summary of what the study found
- The introduction should explicitly state hypotheses (if this was not done it should be clear in
the discussion that the hypotheses were post hoc)
- The method should mention all modifications and changes that were made while making the
study (also be cautious of the use of causal language)
- The same words should be used (there should be no gradual aggrandizement of the results)
- If post hoc interpretation was used this should be treated and mentioned as such
- It should not capitalize on chance findings (by engaging in p-hacking, data mining, or fishing
for significance)
- When the results of the data reveal that the assumptions of statistical techniques were
violated there should be considerations for alternatives, meaning the issue should not be
disregarded
, Lebel (2011): Fearing the Future of Empirical Psychology – Bem’s (2011) Evidence of
Psi as a Case Study of Deficiencies in Modal Research Practice (MRP)
Consequences of deficiences in MRP
- Contributes to weak method-relevant beliefs that lead to a systemic bias in the
interpretation of data that favor’s a researcher’s theory
o Interpretation bias = a bias toward interpretations of data that favor a researcher’s
theory both when the null hypothesis is statistically rejected and not, which can lead
to reporting false positive and disregarding true negatives, thus drawing incorrect
conclusions about human psychology
Can lead to the interpretation of confirmatory data as theory relevant
Can lead to the interpretation of disconfirmatory data as method relevant
- Conservatism (in theory choice) = choosing the theoretical explanation consistent with the
data that requires the least amount of restructuring of the existing knowledge system, thus
reducing ambiguity in the interpretation of data
o The combination of peripheral method-relevant beliefs and central theory-relevant
beliefs is that conservatism becomes an unconditional bias toward interpretations of
data that favor the researcher’s theory
o Often method-relevant beliefs are too peripheral, making them easy to reject
The more it appears that a theoretical explanation has to be the case, the
more likely it is that disconfirming data will be attributed to methodological
flaws
o Often theory-relevant beliefs are too central, making them difficult to reject
Can be weakened by making sure that the hypotheses are falsifiable
Important deficiences in MRP
- Overemphasis on conceptual rather than close replication
o Weakens method-relevant beliefs
o A successful extension becomes a conceptual replication
o A failed extension becomes a methodologically flawed pilot study (which belongs in
the file drawer)
o Defending conceptual replication: one may argue that it can be tricky to get a
procedure just right (however it is a weak warrant for the inflation of the Type l
error) and that in many high-impact papers the reporting of several successful
conceptual replications can be seen as more compelling than several successful close
replications (this is only true if the successful replications were not achieved at the
expense of many failed pilot studies)
- Insufficient attention to verifying the soundness and integrity of measurement and
experimental procedures
o Weakens method-relevant beliefs
o Increases ambiguity in the interpretation of negative (and even positive) results
o Due to the inconsistent, informal, and arduous nature of verifying the integrity of
manipulation and measurement procedures
- Flawed implementation of null hypothesis significance testing (NHST)
o The standard null hypothesis of no difference will almost always be false
It claims that the population means are identical, which is a weak hypothesis
as differences between different populations are inevitable
o Divides theory choice from the context of the broader scientific knowledge system
o Statistical significance is usually treated as the sole criterion of theory choice when
interpreting new data, thereby ignoring all other evidence relevant to the
interpretation of those data
Recommendations for improving MRP
- Stronger emphasis on close replication
the Research Report
Critical thinking = self-directed, self-disciplined, self-monitored, and self-corrective thinking
- Actively anticipating what one expects to read
- Using explicit rules to judge whether the expectations are met
- Constantly monitoring if one’s expectations are being not
- Having clear and justifiable reasons when one makes an evaluative judgement about the
strengths and limitations of an article
Deciding what is right or true is based on several factors
- Habit or tenacity: a state of certainty is preferred over an unsettled state of mind that
accompanies doubt
- Authority: taking judgment of one or more authorities as sufficient evidence
- Democratic judgment: trusting in the judgment of many (can lead to groupthink)
- Reasoning (deductive or inductive)
- Sensory information: personal experience
- Empirical and experimental methods: combines rationality and observation, and is the best
way to produce knowledge that is unbiased
- Reverence for the written word
Interpreting effect sizes
- Almost any effect size can be made to look large or small depending on the choice o the
contrasting effect sizes
- One must decide whether the contrasts were chosen well
Red flags in research reports
- The abstract should not present a false or overly rosy summary of what the study found
- The introduction should explicitly state hypotheses (if this was not done it should be clear in
the discussion that the hypotheses were post hoc)
- The method should mention all modifications and changes that were made while making the
study (also be cautious of the use of causal language)
- The same words should be used (there should be no gradual aggrandizement of the results)
- If post hoc interpretation was used this should be treated and mentioned as such
- It should not capitalize on chance findings (by engaging in p-hacking, data mining, or fishing
for significance)
- When the results of the data reveal that the assumptions of statistical techniques were
violated there should be considerations for alternatives, meaning the issue should not be
disregarded
, Lebel (2011): Fearing the Future of Empirical Psychology – Bem’s (2011) Evidence of
Psi as a Case Study of Deficiencies in Modal Research Practice (MRP)
Consequences of deficiences in MRP
- Contributes to weak method-relevant beliefs that lead to a systemic bias in the
interpretation of data that favor’s a researcher’s theory
o Interpretation bias = a bias toward interpretations of data that favor a researcher’s
theory both when the null hypothesis is statistically rejected and not, which can lead
to reporting false positive and disregarding true negatives, thus drawing incorrect
conclusions about human psychology
Can lead to the interpretation of confirmatory data as theory relevant
Can lead to the interpretation of disconfirmatory data as method relevant
- Conservatism (in theory choice) = choosing the theoretical explanation consistent with the
data that requires the least amount of restructuring of the existing knowledge system, thus
reducing ambiguity in the interpretation of data
o The combination of peripheral method-relevant beliefs and central theory-relevant
beliefs is that conservatism becomes an unconditional bias toward interpretations of
data that favor the researcher’s theory
o Often method-relevant beliefs are too peripheral, making them easy to reject
The more it appears that a theoretical explanation has to be the case, the
more likely it is that disconfirming data will be attributed to methodological
flaws
o Often theory-relevant beliefs are too central, making them difficult to reject
Can be weakened by making sure that the hypotheses are falsifiable
Important deficiences in MRP
- Overemphasis on conceptual rather than close replication
o Weakens method-relevant beliefs
o A successful extension becomes a conceptual replication
o A failed extension becomes a methodologically flawed pilot study (which belongs in
the file drawer)
o Defending conceptual replication: one may argue that it can be tricky to get a
procedure just right (however it is a weak warrant for the inflation of the Type l
error) and that in many high-impact papers the reporting of several successful
conceptual replications can be seen as more compelling than several successful close
replications (this is only true if the successful replications were not achieved at the
expense of many failed pilot studies)
- Insufficient attention to verifying the soundness and integrity of measurement and
experimental procedures
o Weakens method-relevant beliefs
o Increases ambiguity in the interpretation of negative (and even positive) results
o Due to the inconsistent, informal, and arduous nature of verifying the integrity of
manipulation and measurement procedures
- Flawed implementation of null hypothesis significance testing (NHST)
o The standard null hypothesis of no difference will almost always be false
It claims that the population means are identical, which is a weak hypothesis
as differences between different populations are inevitable
o Divides theory choice from the context of the broader scientific knowledge system
o Statistical significance is usually treated as the sole criterion of theory choice when
interpreting new data, thereby ignoring all other evidence relevant to the
interpretation of those data
Recommendations for improving MRP
- Stronger emphasis on close replication
