Responsible Research in Practice
Lecture 1 – Lifecycle Part 1: Generate and Specify Hypotheses ......................................... 2
Week 1 Literature .................................................................................................................... 4
Replicability, Robustness, and Reproducibility in Psychological Science – Nosek., 2022 ................4
What we can and can’t learn from the Many Labs Replication Project – 2013 ................................10
Lecture 2 – Lifecycle Part 2: Design the Study ................................................................... 11
Week 2 Literature .................................................................................................................. 15
Pre-registration in social psychology– van ‘t Veer, 2016 .................................................................15
What’s next for Registered Reports? – Chambers, 2019 ..................................................................18
Lecture 3 – Lifecycle Part 3: Conduct Study and Collect Data ......................................... 20
Week 3 Literature .................................................................................................................. 27
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows
Presenting Anything as Significant – Simmons, 2011 ......................................................................27
Practical Tips for Ethical Data Sharing – Meyer, 2017 ....................................................................28
Lecture 4 – Lifecycle Part 4: Analyze Data & Test Hypotheses ........................................ 31
Week 4 Literature .................................................................................................................. 38
The garden of forking paths: Why multiple comparisons can be a problem, even when there is no
fishing expedition or p-hacking and the research hypothesis was posited ahead of time – Gelman &
Loken, 2023 .......................................................................................................................................38
Why you don’t need to adjust your alpha level for all tests you’ll do in your lifetime – Lakens,
2016 ...................................................................................................................................................40
Lecture 5 – Lifecycle Part 5: Interpret data ........................................................................ 42
Week 5 Literature .................................................................................................................. 47
The New Statistics: Why and How – Cumming, 2013 .....................................................................47
Fooling ourselves – Nuzzo, 2015 ......................................................................................................50
Lecture 6 – Lifecycle Part 6: Publish or conduct next study ............................................. 51
Week 6 Literature .................................................................................................................. 55
Ten common statistical mistakes to watch out for when writing or reviewing a manyscript – Makin,
2019 ...................................................................................................................................................55
Fallibility in Science: Responding to Errors in the Work of Oneself and Others – Bishop, 2018 ....56
,Objectives:
• Critically assessing own research practices and those of others
• Analysing strengths and weaknesses in the stages of the research lifecycle from a
research integrity perspective
• Identifying and explaining methodological, meta-scientific, and statistical practices
that go beyond standard practices in psychology
Exam:
• Covers general aspects that are discussed in the lectures → prepare by studying the
slides, reading the literature, and lecture notes
• Test → ask yourself if you are able to explain something about each slide
• Open ended questions
Assignments
1. Review report → report about the interview, should state the questions you asked, the
answers obtained, and add a critical evaluation of the answers (is it good or bad
practice, and why?). The report also includes a recommendation of another analysis!
Deadline 23rd may, 13:00. Max 5 pages (1.5 spacing). Give the full list of
questions in the appendix.
2. Reply to the alternative → perform the alternative analysis that is suggested by the
reviewer, explain pros and cons of each approach, compare the analyses and explain
which one you pick and why. Deadline 6th June. Max 2 pages
1
,Lecture 1 – Lifecycle Part 1: Generate and Specify Hypotheses
Today’s lecture will entail:
• Integrity of existing research
• Reproducibility, replicability and robustness
• Specifying hypotheses
Integrity of existing research
There are different components to this → the values and behaviours of researchers (there
usually is a code of conduct), but also the integrity of the literature.
Why do we have codes, education etc. The codes don’t cover everything of course, as they do
not apply to all complex day-to-day situations. But it’s still good to have them, for education,
continued calibration of moral compass, community. It is for example about how to
communicate, report, or share data. It’s about professional standards (e.g., for research
integrity) and it is updated over time and develops with other changes in the system.
Mentorship and leadership are the way we transfer rules, and this document helps with that.
Also, reflect on it, which will make it easier to react when future dilemmas arise. The codes
also help with starting conversations about how you personally, and as part of a team, relate
to the topics we discuss.
Responsible scholarship → conducting our work with integrity, and meeting the needs for
better quality and efficiency in psychological science. Related to open science and ethics,
which are related to emancipation and social safety.
Why does responsible scholarship matter:
• It safeguards the quality and progress of science and its application
• It enables trust among members of the scientific community
• It safeguards the reputation of science
• Fosters equal opportunities and outcomes → equity
• It prevents research waste (research is done with money of taxpayers)
• Build a robust cumulative psychological science
Goal of science → UNESCO → by promoting science that is more accessible, inclusive and
transparent, open science furthers the right of everyone to share scientific advancement and
its benefits. The goals include benefitting society by advancing knowledge in a credible and
unbiased manner, for which norms such as transparency, honesty, diversity, and equity are
important.
But there are also pressures in the system that drive the goals of science apart →
competitiveness, hectic pace, publish or perish, external funding or expulsion. This moves the
individual to selfishness, secrecy, valuing quantity over quality, and a range of irresponsible
practices.
There are the goals of the individual, and the goals of science. The more they overlap, the
more intrinsic motivation an individual has towards science.
How do we live up to the goal? Science is self-correcting. Openness and transparency will
facilitate this self-correction!
2
, Examples of dilemmas PhD candidates deal with → being tempted to present things prettier,
HARKing, optional stopping, selective reporting, honorary authorships, not standing behind
work but being pressured to publish, throwing away the protocol because the supervisor
doesn’t want to commit, adding up to threatening mental health.
Progress is made by small improvements to culture. Take into account context and culture.
Research culture change needs all levels of intervention. At the personal level, we can
cultivate our own compass to navigate this landscape. Direct influence on peers, indirect
influence of behaviours.
You don’t have to do these in a chronological order. It is done with small improvements. So,
the rigor-enhancing categories are infrastructure, user experience, communities,
incentives and policy.
Distortions to integrity of psychology:
• Scientific misconduct (Diederik Stapel)
• Questionable research practice
• Poor research practice (incompetence)
• Honest errors and mistakes (fallibility)
Daily grey areas are problematic but also the spot we can change the most.
Integrity of existing work → all the methods and practices of researcher together make up
the integrity of the literature. What can we tell about previous work we build on?
Reproducibility, replicability, and robustness.
Replication → ‘many labs’ projects → a study that has been done in many different labs.
Large scale replication projects revealed that there is a problem. Large scale collaborations
and independent replications can work towards solving the problem. We need to invest
3
Lecture 1 – Lifecycle Part 1: Generate and Specify Hypotheses ......................................... 2
Week 1 Literature .................................................................................................................... 4
Replicability, Robustness, and Reproducibility in Psychological Science – Nosek., 2022 ................4
What we can and can’t learn from the Many Labs Replication Project – 2013 ................................10
Lecture 2 – Lifecycle Part 2: Design the Study ................................................................... 11
Week 2 Literature .................................................................................................................. 15
Pre-registration in social psychology– van ‘t Veer, 2016 .................................................................15
What’s next for Registered Reports? – Chambers, 2019 ..................................................................18
Lecture 3 – Lifecycle Part 3: Conduct Study and Collect Data ......................................... 20
Week 3 Literature .................................................................................................................. 27
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows
Presenting Anything as Significant – Simmons, 2011 ......................................................................27
Practical Tips for Ethical Data Sharing – Meyer, 2017 ....................................................................28
Lecture 4 – Lifecycle Part 4: Analyze Data & Test Hypotheses ........................................ 31
Week 4 Literature .................................................................................................................. 38
The garden of forking paths: Why multiple comparisons can be a problem, even when there is no
fishing expedition or p-hacking and the research hypothesis was posited ahead of time – Gelman &
Loken, 2023 .......................................................................................................................................38
Why you don’t need to adjust your alpha level for all tests you’ll do in your lifetime – Lakens,
2016 ...................................................................................................................................................40
Lecture 5 – Lifecycle Part 5: Interpret data ........................................................................ 42
Week 5 Literature .................................................................................................................. 47
The New Statistics: Why and How – Cumming, 2013 .....................................................................47
Fooling ourselves – Nuzzo, 2015 ......................................................................................................50
Lecture 6 – Lifecycle Part 6: Publish or conduct next study ............................................. 51
Week 6 Literature .................................................................................................................. 55
Ten common statistical mistakes to watch out for when writing or reviewing a manyscript – Makin,
2019 ...................................................................................................................................................55
Fallibility in Science: Responding to Errors in the Work of Oneself and Others – Bishop, 2018 ....56
,Objectives:
• Critically assessing own research practices and those of others
• Analysing strengths and weaknesses in the stages of the research lifecycle from a
research integrity perspective
• Identifying and explaining methodological, meta-scientific, and statistical practices
that go beyond standard practices in psychology
Exam:
• Covers general aspects that are discussed in the lectures → prepare by studying the
slides, reading the literature, and lecture notes
• Test → ask yourself if you are able to explain something about each slide
• Open ended questions
Assignments
1. Review report → report about the interview, should state the questions you asked, the
answers obtained, and add a critical evaluation of the answers (is it good or bad
practice, and why?). The report also includes a recommendation of another analysis!
Deadline 23rd may, 13:00. Max 5 pages (1.5 spacing). Give the full list of
questions in the appendix.
2. Reply to the alternative → perform the alternative analysis that is suggested by the
reviewer, explain pros and cons of each approach, compare the analyses and explain
which one you pick and why. Deadline 6th June. Max 2 pages
1
,Lecture 1 – Lifecycle Part 1: Generate and Specify Hypotheses
Today’s lecture will entail:
• Integrity of existing research
• Reproducibility, replicability and robustness
• Specifying hypotheses
Integrity of existing research
There are different components to this → the values and behaviours of researchers (there
usually is a code of conduct), but also the integrity of the literature.
Why do we have codes, education etc. The codes don’t cover everything of course, as they do
not apply to all complex day-to-day situations. But it’s still good to have them, for education,
continued calibration of moral compass, community. It is for example about how to
communicate, report, or share data. It’s about professional standards (e.g., for research
integrity) and it is updated over time and develops with other changes in the system.
Mentorship and leadership are the way we transfer rules, and this document helps with that.
Also, reflect on it, which will make it easier to react when future dilemmas arise. The codes
also help with starting conversations about how you personally, and as part of a team, relate
to the topics we discuss.
Responsible scholarship → conducting our work with integrity, and meeting the needs for
better quality and efficiency in psychological science. Related to open science and ethics,
which are related to emancipation and social safety.
Why does responsible scholarship matter:
• It safeguards the quality and progress of science and its application
• It enables trust among members of the scientific community
• It safeguards the reputation of science
• Fosters equal opportunities and outcomes → equity
• It prevents research waste (research is done with money of taxpayers)
• Build a robust cumulative psychological science
Goal of science → UNESCO → by promoting science that is more accessible, inclusive and
transparent, open science furthers the right of everyone to share scientific advancement and
its benefits. The goals include benefitting society by advancing knowledge in a credible and
unbiased manner, for which norms such as transparency, honesty, diversity, and equity are
important.
But there are also pressures in the system that drive the goals of science apart →
competitiveness, hectic pace, publish or perish, external funding or expulsion. This moves the
individual to selfishness, secrecy, valuing quantity over quality, and a range of irresponsible
practices.
There are the goals of the individual, and the goals of science. The more they overlap, the
more intrinsic motivation an individual has towards science.
How do we live up to the goal? Science is self-correcting. Openness and transparency will
facilitate this self-correction!
2
, Examples of dilemmas PhD candidates deal with → being tempted to present things prettier,
HARKing, optional stopping, selective reporting, honorary authorships, not standing behind
work but being pressured to publish, throwing away the protocol because the supervisor
doesn’t want to commit, adding up to threatening mental health.
Progress is made by small improvements to culture. Take into account context and culture.
Research culture change needs all levels of intervention. At the personal level, we can
cultivate our own compass to navigate this landscape. Direct influence on peers, indirect
influence of behaviours.
You don’t have to do these in a chronological order. It is done with small improvements. So,
the rigor-enhancing categories are infrastructure, user experience, communities,
incentives and policy.
Distortions to integrity of psychology:
• Scientific misconduct (Diederik Stapel)
• Questionable research practice
• Poor research practice (incompetence)
• Honest errors and mistakes (fallibility)
Daily grey areas are problematic but also the spot we can change the most.
Integrity of existing work → all the methods and practices of researcher together make up
the integrity of the literature. What can we tell about previous work we build on?
Reproducibility, replicability, and robustness.
Replication → ‘many labs’ projects → a study that has been done in many different labs.
Large scale replication projects revealed that there is a problem. Large scale collaborations
and independent replications can work towards solving the problem. We need to invest
3
