Responsible Research in Practice -
Summary
, Lectures
Week 1: Building on previous work - Generating and specifying
hypotheses
Integrity of existing research
- Values and behaviour of the researchers
- Are they being responsible, honest, and ethical (for example look at code of
conduct)?
- Integrity of literature
- Do they present knowledge properly, are they reliable, good, which ones do we
find, read, and use?
- When you write and read, think about whether you would be able to replicate the
study (aka be responsible and transparent)
- Who are the authors and what might have been there or their funders agenda?
- When you read the methods and results, what would you conclude
- It is also difficult to be an expert on everything: maybe collaborate
- We are generally very subject to biases and there are many different biases
- Help yourself be proof against those biases. E.g., ask others and what
they would do etc.
- Reading scientific reports
- Published reports are the basis, so we are aware ere is a bias in them
- Researchers are people, people have biases
- R the understanding of papers t is important to be critical
Reproducibility and replicability
- Lately, there was a much bigger effort to try and see whether studies replicate
- There are not actually that many that can be replicated
- Large scale replication
- Replication project reveals that there is a problem
- large-scale collaborations and independent replication can work towards solving
this problem
- We need to invest more in figuring out what replication means and how to do it
- Re replications are needed, but how do we select what is worthy of replicating?
- Example: PhD students as a basis for further research
- Many people say that there is a crisis, but it is not necessarily specific to
psychology
- How to overcome some difficulties and prevent lack of replicability
- Reproducible and transparent
- Make sure you are very clear and open about what you did
- For example, share your data, clear methods, share code etc
- Large-scale replications
- Websites and places where researchers can upload their data etc.
- When people share their data, it often becomes better, plus it can be checked by
others
, - Websites also help find studies that are in line with certain findings, mentioned or
contradicting
- Planning of the research
- Before collecting data, make a plan. Can include: RQ, hypotheses, population,
sample size, etc. etc.
Specifying hypotheses
- Make specific and testable hypotheses before you collect data. Best if you make them
quantifiable (to what extent). Additional: make a minimal effect of interest
- Example Harking: making hypotheses after results are known. So, using your data
twice. You cannot prove anything with this. You can get an idea on one part and test it in
another, or make clear that it is exploratory etc.
- The less responsible you are to your research, the less likely it becomes to be replicated
(responsibly)
Week 2: Design study
- Think about what you will do before what you will do. Write out your intentions and what
you will conclude when you get certain outcomes. And what will you do when you get
certain conclusions? Also, think about what your bigger goals are when you conduct a
study. Is it worth it, what are the potential benefits or drawbacks?
Sampling from Population
- If you have the entire population then you do not really need any tests because you can
see the real numbers for the population. You only need tests if you are trying to
generalise it for people outside the sample you have
- Variation:
- Be aware of Measurement error and how your statistical model deals with that
- Other sauces of error: random sampling of the population & random allocation of
treatments
- Sample
- The population must be clearly defined before eth sampling
- Perform a power analysis to inform yourself about the sample size
- Sample size also depends on how you sample it and who you sample (for
example families etc)
- Are the elements drawn at random?
- Re the elements drawn independently
- If only one or none are present, then you can still make inferences but
only for that sample and not necessarily for the rest of the population
- Therefore: also, be careful how your state your conclusions and to what degree
you can generalize your statements
Determining Sample Size
- There is no one universal way or rule of thumb to determine the sample size
- It is an interplay of the effect size, significant level, power, precision
(more than .05 for alpha (more Type 1 error) → okay to have smaller
sample size; more Type 2 error → okay to decrease sample size (less
power)
Summary
, Lectures
Week 1: Building on previous work - Generating and specifying
hypotheses
Integrity of existing research
- Values and behaviour of the researchers
- Are they being responsible, honest, and ethical (for example look at code of
conduct)?
- Integrity of literature
- Do they present knowledge properly, are they reliable, good, which ones do we
find, read, and use?
- When you write and read, think about whether you would be able to replicate the
study (aka be responsible and transparent)
- Who are the authors and what might have been there or their funders agenda?
- When you read the methods and results, what would you conclude
- It is also difficult to be an expert on everything: maybe collaborate
- We are generally very subject to biases and there are many different biases
- Help yourself be proof against those biases. E.g., ask others and what
they would do etc.
- Reading scientific reports
- Published reports are the basis, so we are aware ere is a bias in them
- Researchers are people, people have biases
- R the understanding of papers t is important to be critical
Reproducibility and replicability
- Lately, there was a much bigger effort to try and see whether studies replicate
- There are not actually that many that can be replicated
- Large scale replication
- Replication project reveals that there is a problem
- large-scale collaborations and independent replication can work towards solving
this problem
- We need to invest more in figuring out what replication means and how to do it
- Re replications are needed, but how do we select what is worthy of replicating?
- Example: PhD students as a basis for further research
- Many people say that there is a crisis, but it is not necessarily specific to
psychology
- How to overcome some difficulties and prevent lack of replicability
- Reproducible and transparent
- Make sure you are very clear and open about what you did
- For example, share your data, clear methods, share code etc
- Large-scale replications
- Websites and places where researchers can upload their data etc.
- When people share their data, it often becomes better, plus it can be checked by
others
, - Websites also help find studies that are in line with certain findings, mentioned or
contradicting
- Planning of the research
- Before collecting data, make a plan. Can include: RQ, hypotheses, population,
sample size, etc. etc.
Specifying hypotheses
- Make specific and testable hypotheses before you collect data. Best if you make them
quantifiable (to what extent). Additional: make a minimal effect of interest
- Example Harking: making hypotheses after results are known. So, using your data
twice. You cannot prove anything with this. You can get an idea on one part and test it in
another, or make clear that it is exploratory etc.
- The less responsible you are to your research, the less likely it becomes to be replicated
(responsibly)
Week 2: Design study
- Think about what you will do before what you will do. Write out your intentions and what
you will conclude when you get certain outcomes. And what will you do when you get
certain conclusions? Also, think about what your bigger goals are when you conduct a
study. Is it worth it, what are the potential benefits or drawbacks?
Sampling from Population
- If you have the entire population then you do not really need any tests because you can
see the real numbers for the population. You only need tests if you are trying to
generalise it for people outside the sample you have
- Variation:
- Be aware of Measurement error and how your statistical model deals with that
- Other sauces of error: random sampling of the population & random allocation of
treatments
- Sample
- The population must be clearly defined before eth sampling
- Perform a power analysis to inform yourself about the sample size
- Sample size also depends on how you sample it and who you sample (for
example families etc)
- Are the elements drawn at random?
- Re the elements drawn independently
- If only one or none are present, then you can still make inferences but
only for that sample and not necessarily for the rest of the population
- Therefore: also, be careful how your state your conclusions and to what degree
you can generalize your statements
Determining Sample Size
- There is no one universal way or rule of thumb to determine the sample size
- It is an interplay of the effect size, significant level, power, precision
(more than .05 for alpha (more Type 1 error) → okay to have smaller
sample size; more Type 2 error → okay to decrease sample size (less
power)
