Class 1
Pride and Prejudice: Psychological Insights into the Replication Crisis
Purpose
- Foster critical thinking regarding scientific output
- Conduct better original research
- Adapt responsible research/open science principles.
Practical: Plan a replication of an existing study.
Research
- Develop an idea
- Design study
- Peer review
- Collect and analyze data
- Write report
- Peer report
- Publish your report
3 Categories of Scientific Misconduct
- Fabrication: making up data
- Falsification: distorting data
- Questionable research practices: cooking data, mining data, concealing data.
More of a gray area because scientists might not be aware that they are doing
something wrong.
Often the result of hidden biases, carelessness or lack of training.
Misrepresenting the participants words.
Selectively reporting the studies that worked.
Reporting an unexpected finding as having been predicted from the start
(HARKing)
Deciding to exclude the data after looking at the impact of it on the results.
Regular misbehaviors may present a greater threat to science than fabrication,
and plagiarism.
Purpose of replication
- Psychologists are often proud of themselves for the methods they use
- Our methods set us aside from other, related disciplines.
- Our methods might be susceptible to bias, prejudice and fraud.
- We should critically evaluate our methods
- Not limited to psychology.
- Psychology might offer insights into bias, prejudice and fraud.
- Objectivity is the essence of science.
David Marx
- The social professor that had two professors that had committed fraud (both the
professor at the Harvard and Diederik).
- His PHD students were generally able to replicate the results he found with Diederik.
- Diederik had typed out data that confirmed his hypothesis, threw the empty data to the
dumpster.
, Fraud
- Can go unnoticed by co-authors, reviewers and scientific community.
- Psychology is not more vulnerable than any other sciences.
- 3/40 fraud cases of Diederik was detected by replication, 3/40 was found by peer
review. 21/40 was discovered by whistle-blowers.
- Fraud is more prevalent now. Data is kept online. It is easy to change or add data.
- Psychology is a behavior science. When a study cannot be replicated, the original
author can easily defend themselves by saying that people and behaviors are different.
- Large number of science practitioners.
- Incentive structure: more emphasis and importance on the number of articles you
produce not the quality of the articles.
- Absence of systematic replications: not many people are engaging in replications.
Consequences of misconduct
- Misuse of tax payers’ money: no return or investment.
- Huge waste of energy: collaborators, reviewers, scientific community at large.
- Reputation damage: researchers, universities, discipline.
- Salami slice method: It is now frowned upon to conduct one study with three
hypothesis and publish three separate papers from one study.
- Gross reporting error: when a researcher claims that the p value is lower than .05,
claiming that there is a statistical difference between condition, whereas the reality is
that the p value is greater than .05.
- Retraction: a scientific journal issued a statement that they have found a problem and
announcing that the publication should be removed.
Class 2
- When given an opportunity, people usually cheat during the experiments.
- The experimenter should have some sort of way to check whether if the participant is cheating or
whether if they are answering in an honest manner.
- There are handful of extreme cheaters. However, people who cheat a lot less, is way more
abundant.
Recommendations
- Educate about ethical behavior and fraud.
- Audits by grant providers.
- Thoroughly follow-up suspicions of fraud.
- Detailed reports about data collection and analysis to be able to replicate studies.
- Make data available for reproduction.
- Pre-registration to explain what you will be doing in your research and how.
Research Cycle
- Specify hypothesis (lack of replication, researchers would like to check hypothesis that might be
supported by evidence)
- Design study
- Collect data
- Analyze data
- Interpret data
- Publish experiment (publication bias and lack of data share)
Publication Bias
- Outcome of a study influences the decisions to publish it
- Both journals and researchers want surprising, counterintuitive, headline-making findings
Pride and Prejudice: Psychological Insights into the Replication Crisis
Purpose
- Foster critical thinking regarding scientific output
- Conduct better original research
- Adapt responsible research/open science principles.
Practical: Plan a replication of an existing study.
Research
- Develop an idea
- Design study
- Peer review
- Collect and analyze data
- Write report
- Peer report
- Publish your report
3 Categories of Scientific Misconduct
- Fabrication: making up data
- Falsification: distorting data
- Questionable research practices: cooking data, mining data, concealing data.
More of a gray area because scientists might not be aware that they are doing
something wrong.
Often the result of hidden biases, carelessness or lack of training.
Misrepresenting the participants words.
Selectively reporting the studies that worked.
Reporting an unexpected finding as having been predicted from the start
(HARKing)
Deciding to exclude the data after looking at the impact of it on the results.
Regular misbehaviors may present a greater threat to science than fabrication,
and plagiarism.
Purpose of replication
- Psychologists are often proud of themselves for the methods they use
- Our methods set us aside from other, related disciplines.
- Our methods might be susceptible to bias, prejudice and fraud.
- We should critically evaluate our methods
- Not limited to psychology.
- Psychology might offer insights into bias, prejudice and fraud.
- Objectivity is the essence of science.
David Marx
- The social professor that had two professors that had committed fraud (both the
professor at the Harvard and Diederik).
- His PHD students were generally able to replicate the results he found with Diederik.
- Diederik had typed out data that confirmed his hypothesis, threw the empty data to the
dumpster.
, Fraud
- Can go unnoticed by co-authors, reviewers and scientific community.
- Psychology is not more vulnerable than any other sciences.
- 3/40 fraud cases of Diederik was detected by replication, 3/40 was found by peer
review. 21/40 was discovered by whistle-blowers.
- Fraud is more prevalent now. Data is kept online. It is easy to change or add data.
- Psychology is a behavior science. When a study cannot be replicated, the original
author can easily defend themselves by saying that people and behaviors are different.
- Large number of science practitioners.
- Incentive structure: more emphasis and importance on the number of articles you
produce not the quality of the articles.
- Absence of systematic replications: not many people are engaging in replications.
Consequences of misconduct
- Misuse of tax payers’ money: no return or investment.
- Huge waste of energy: collaborators, reviewers, scientific community at large.
- Reputation damage: researchers, universities, discipline.
- Salami slice method: It is now frowned upon to conduct one study with three
hypothesis and publish three separate papers from one study.
- Gross reporting error: when a researcher claims that the p value is lower than .05,
claiming that there is a statistical difference between condition, whereas the reality is
that the p value is greater than .05.
- Retraction: a scientific journal issued a statement that they have found a problem and
announcing that the publication should be removed.
Class 2
- When given an opportunity, people usually cheat during the experiments.
- The experimenter should have some sort of way to check whether if the participant is cheating or
whether if they are answering in an honest manner.
- There are handful of extreme cheaters. However, people who cheat a lot less, is way more
abundant.
Recommendations
- Educate about ethical behavior and fraud.
- Audits by grant providers.
- Thoroughly follow-up suspicions of fraud.
- Detailed reports about data collection and analysis to be able to replicate studies.
- Make data available for reproduction.
- Pre-registration to explain what you will be doing in your research and how.
Research Cycle
- Specify hypothesis (lack of replication, researchers would like to check hypothesis that might be
supported by evidence)
- Design study
- Collect data
- Analyze data
- Interpret data
- Publish experiment (publication bias and lack of data share)
Publication Bias
- Outcome of a study influences the decisions to publish it
- Both journals and researchers want surprising, counterintuitive, headline-making findings
