SSR II
Critical thinking about scientific reasoning
Biases seen in the scientific field:
1. Publication bias: This refers to the tendency of journals and
publishers to favor studies with positive, significant, or novel results
over those with null or negative findings
2. File drawer problem: This is closely related to publication bias and
refers to the practice of "hiding away" studies with non-significant or
negative results, meaning they never get published or shared
a. 50% (estimate) of studies in psychology remain unpublished
b. Too much emphasis on new, surprising, findings is problematic
i. Overemphasis on counterintuitive findings and small
and noisy samples
1. Counterintuitive has a low base rate (the results
haven’t happen a lot)
a. Even with an alpha with .05, the probability
that you reject the null hypotheses while it is
true, is 50/50
2. Small and noisy samples, make the result vary a
lot (unreliable), making it easier to find a result
ii. Popular media will highlight catchy findings, and thinking
that a positive effect result in a full-fledged phenomenon
3. Unspecified predictions: This stands for Hypothesizing After
the Results are Known and refers to creating or altering
hypotheses based on the observed data, rather than sticking to the
original predictions
a. HARKing
4. Hidden unsustainable assumptions in theory: These are
unexamined or unrealistic assumptions in a theory or model that, if
challenged, would undermine the results or conclusions
5. The texas sharpshooter fallacy: This fallacy occurs when
someone cherry-picks data or a pattern to fit a preconceived
conclusion, ignoring information that might contradict or
complicate the conclusion
6. Just-so storytelling: Refers to giving a plausible but
unfalsifiable or overly simplistic explanation for something,
often without enough evidence or ignoring other possible
explanations
Thinking research (e.g., the effect of medicine) backs up the theory due
to:
1. Publication bias
2. Outcome reporting bias: S
, 3. Spin: In research, "spin" refers to the practice of presenting or
interpreting study results in a way that makes them appear more
favorable or conclusive than they really are
4. Citation bias: Citation bias occurs when researchers preferentially
cite studies that support their own findings or viewpoints, while
ignoring studies with conflicting or null results
Important aspects of a research:
1. Replicability:
a. Research needs to be replicable with a different sample
b. Getting different results (when replicating) shows bad research
i. But some difference can be explained by the different
samples
c. Different methods can also be used to see whether the results
are consistent
d. Replication crisis: Well-known studies have failed to
replicate
2. Robustness: Refers to the stability and reliability of research
findings across different analyses, models, or assumptions. It
assesses whether the results hold true when the conditions of the
experiment or analysis are varied
3. Reproducibility: To what extent others can reproduce the same
data by using/doing the exact same method/analysis
Researcher degrees of freedom:
Degree of freedom for a researcher: The various choices and decisions
researchers can make during the design, data collection, analysis, and
reporting phases of a study
- Ideal: Peer reviews are correct, no biases used
- Real: Some articles are not published because it does not show any
significant findings, biases are used when reviewing
The goals of science and of the scientist might not align
- Quantity of (frequently quotes) publications essential for further
careers
o Making scientist lie about their significance to be able to
publish their research and keep their job
o Report bias: This occurs when researchers selectively report
certain aspects of their study (e.g., significant results) while
downplaying or omitting non-significant or unfavorable
findings
- The problem is that the pursuit of truthful/robust and
interpretable results in the current system does not always lead
to publications
o Publication bias
Understanding questionable research practices
- P-hacking
Critical thinking about scientific reasoning
Biases seen in the scientific field:
1. Publication bias: This refers to the tendency of journals and
publishers to favor studies with positive, significant, or novel results
over those with null or negative findings
2. File drawer problem: This is closely related to publication bias and
refers to the practice of "hiding away" studies with non-significant or
negative results, meaning they never get published or shared
a. 50% (estimate) of studies in psychology remain unpublished
b. Too much emphasis on new, surprising, findings is problematic
i. Overemphasis on counterintuitive findings and small
and noisy samples
1. Counterintuitive has a low base rate (the results
haven’t happen a lot)
a. Even with an alpha with .05, the probability
that you reject the null hypotheses while it is
true, is 50/50
2. Small and noisy samples, make the result vary a
lot (unreliable), making it easier to find a result
ii. Popular media will highlight catchy findings, and thinking
that a positive effect result in a full-fledged phenomenon
3. Unspecified predictions: This stands for Hypothesizing After
the Results are Known and refers to creating or altering
hypotheses based on the observed data, rather than sticking to the
original predictions
a. HARKing
4. Hidden unsustainable assumptions in theory: These are
unexamined or unrealistic assumptions in a theory or model that, if
challenged, would undermine the results or conclusions
5. The texas sharpshooter fallacy: This fallacy occurs when
someone cherry-picks data or a pattern to fit a preconceived
conclusion, ignoring information that might contradict or
complicate the conclusion
6. Just-so storytelling: Refers to giving a plausible but
unfalsifiable or overly simplistic explanation for something,
often without enough evidence or ignoring other possible
explanations
Thinking research (e.g., the effect of medicine) backs up the theory due
to:
1. Publication bias
2. Outcome reporting bias: S
, 3. Spin: In research, "spin" refers to the practice of presenting or
interpreting study results in a way that makes them appear more
favorable or conclusive than they really are
4. Citation bias: Citation bias occurs when researchers preferentially
cite studies that support their own findings or viewpoints, while
ignoring studies with conflicting or null results
Important aspects of a research:
1. Replicability:
a. Research needs to be replicable with a different sample
b. Getting different results (when replicating) shows bad research
i. But some difference can be explained by the different
samples
c. Different methods can also be used to see whether the results
are consistent
d. Replication crisis: Well-known studies have failed to
replicate
2. Robustness: Refers to the stability and reliability of research
findings across different analyses, models, or assumptions. It
assesses whether the results hold true when the conditions of the
experiment or analysis are varied
3. Reproducibility: To what extent others can reproduce the same
data by using/doing the exact same method/analysis
Researcher degrees of freedom:
Degree of freedom for a researcher: The various choices and decisions
researchers can make during the design, data collection, analysis, and
reporting phases of a study
- Ideal: Peer reviews are correct, no biases used
- Real: Some articles are not published because it does not show any
significant findings, biases are used when reviewing
The goals of science and of the scientist might not align
- Quantity of (frequently quotes) publications essential for further
careers
o Making scientist lie about their significance to be able to
publish their research and keep their job
o Report bias: This occurs when researchers selectively report
certain aspects of their study (e.g., significant results) while
downplaying or omitting non-significant or unfavorable
findings
- The problem is that the pursuit of truthful/robust and
interpretable results in the current system does not always lead
to publications
o Publication bias
Understanding questionable research practices
- P-hacking
