Scientific and Statistical Reasoning
Lecture Notes Interim 2
Contents
Lecture 11 - Critical Thinking about Psychological Research .................................................................. 2
Lecture 12 - F-distribution & One-Way ANOVA .................................................................................... 15
Lecture 13 - Non-parametric testing ..................................................................................................... 29
Lecture 14 - Critical Thinking about Statistical Inference ...................................................................... 46
Lecture 15 – ANCOVA............................................................................................................................ 61
Lecture 16 - Factorial ANOVA ................................................................................................................ 73
Lecture 17 - Repeated & Mixed ANOVA ................................................................................................ 92
Lecture 18 - Chi-Squared Test ............................................................................................................. 108
Lecture 19 - Bayesian Parameter Estimation & Hypothesis Testing .................................................... 118
,Lecture 11 - Critical Thinking about Psychological Research
"The first principle is that you must not fool yourself - and you are the easiest person to fool.
So you have to be very careful about that. After you've not fooled yourself, it's easy not to fool
other scientists. You just have to be honest in a conventional way after that."
Feyman (1974)
Scientific & Statistical Reasoning
Critical thinking about psychological research
"Disbelief is not an option. The results are not made up, nor are they statistical flukes. You
have no choice but to accept that the major conclusions of these studies are true. More
important, you must accept that they are true about you."
Replicability
𝑝-values in replication studies are a lot more diverse than in the original studies. In the original
studies they're clustered under the critical value.
Same difference with effect size. Effect size in the original studies are a lot higher than in
replication studies.
Robustness
The results from the same studies can differ a lot depending on who (which scientists) does the
study.
Reproducibility
• 49% of papers published in "13 well-regarded economics journals" could be reproduced
(when provided with data and code)
• Half of publications in 8 major psychology journals (1953-2013) that involve NHST contain at
least 1 error in reported 𝑝-values.
o 1 in 8 contained a "grossly inconsistent 𝑝-value that may have affected the statistical
conclusion".
o Those gross inconsistencies were more likely when significant vs non-significant
results were reported
Researcher degrees of freedom
• Which research area, which theory, which hypotheses?
• How many dependent variables, how many conditions?
• What measurement procedure?
• How many participants?
• What analysis? What outliers?
• What is an effect? What is a relevant effect?
• What can you conclude from the analysis?
• What can you conclude from the investigations?
Today
1. Ideal science vs real science
2. Understanding questionable research practices
3. Causes & solutions
,Bending over backwards & making errors
• The first principle is that you must not fool yourself - and you are the easiest person
to fool. So you have to be very careful about that. After you've not fooled yourself, it's easy
not to fool other scientists. You just have to be honest in a conventional way after that.
• (...) I'm not trying to tell you what to do about cheating on your wife, or fooling your
girlfriend, or something like that, when you're not trying to be a scientists, but just trying to
be an ordinary human being. We'll leave those problems up to you and your rabbi. I'm
talking about a specific, extra type of integrity that is not lying, but bending over
backwards to show how you're maybe wrong, that you ought to do when acting as a
scientist. And this is our responsibility as scientists, certainly to other scientists, and I think
to laymen.
• "[the essence of science is] making mistakes in public - making mistakes for all to see, in
the hopes of getting the others to help with the corrections.
If you leave out enough, everything can become a beautiful story.
• Publication bias, file drawer problem
• Unspecified predictions, hidden unsustainable assumptions in theory.
Ideal vs Real
• Ideal scientists:
• Rational
• Unbiased
• Super intelligent
• Fair
• Communicates well
• Makes no mistakes
, Ideal
Real
Reviews are a lot more negative even when the only difference is negative results instead of positive
results.
• A letter from an editor of a major environmental/toxicological journal rejecting a manuscript
because of its negative findings included the statement:
o Unfortunately, we are not able to publish this manuscripts. The manuscript is very
well written and the study was well documented. Unfortunately, the negative
results translates into a minimal contribution to the field. We encourage you to
continue your work in this area and we will be glad to consider additional
manuscripts that you may prepare in the future.
Lecture Notes Interim 2
Contents
Lecture 11 - Critical Thinking about Psychological Research .................................................................. 2
Lecture 12 - F-distribution & One-Way ANOVA .................................................................................... 15
Lecture 13 - Non-parametric testing ..................................................................................................... 29
Lecture 14 - Critical Thinking about Statistical Inference ...................................................................... 46
Lecture 15 – ANCOVA............................................................................................................................ 61
Lecture 16 - Factorial ANOVA ................................................................................................................ 73
Lecture 17 - Repeated & Mixed ANOVA ................................................................................................ 92
Lecture 18 - Chi-Squared Test ............................................................................................................. 108
Lecture 19 - Bayesian Parameter Estimation & Hypothesis Testing .................................................... 118
,Lecture 11 - Critical Thinking about Psychological Research
"The first principle is that you must not fool yourself - and you are the easiest person to fool.
So you have to be very careful about that. After you've not fooled yourself, it's easy not to fool
other scientists. You just have to be honest in a conventional way after that."
Feyman (1974)
Scientific & Statistical Reasoning
Critical thinking about psychological research
"Disbelief is not an option. The results are not made up, nor are they statistical flukes. You
have no choice but to accept that the major conclusions of these studies are true. More
important, you must accept that they are true about you."
Replicability
𝑝-values in replication studies are a lot more diverse than in the original studies. In the original
studies they're clustered under the critical value.
Same difference with effect size. Effect size in the original studies are a lot higher than in
replication studies.
Robustness
The results from the same studies can differ a lot depending on who (which scientists) does the
study.
Reproducibility
• 49% of papers published in "13 well-regarded economics journals" could be reproduced
(when provided with data and code)
• Half of publications in 8 major psychology journals (1953-2013) that involve NHST contain at
least 1 error in reported 𝑝-values.
o 1 in 8 contained a "grossly inconsistent 𝑝-value that may have affected the statistical
conclusion".
o Those gross inconsistencies were more likely when significant vs non-significant
results were reported
Researcher degrees of freedom
• Which research area, which theory, which hypotheses?
• How many dependent variables, how many conditions?
• What measurement procedure?
• How many participants?
• What analysis? What outliers?
• What is an effect? What is a relevant effect?
• What can you conclude from the analysis?
• What can you conclude from the investigations?
Today
1. Ideal science vs real science
2. Understanding questionable research practices
3. Causes & solutions
,Bending over backwards & making errors
• The first principle is that you must not fool yourself - and you are the easiest person
to fool. So you have to be very careful about that. After you've not fooled yourself, it's easy
not to fool other scientists. You just have to be honest in a conventional way after that.
• (...) I'm not trying to tell you what to do about cheating on your wife, or fooling your
girlfriend, or something like that, when you're not trying to be a scientists, but just trying to
be an ordinary human being. We'll leave those problems up to you and your rabbi. I'm
talking about a specific, extra type of integrity that is not lying, but bending over
backwards to show how you're maybe wrong, that you ought to do when acting as a
scientist. And this is our responsibility as scientists, certainly to other scientists, and I think
to laymen.
• "[the essence of science is] making mistakes in public - making mistakes for all to see, in
the hopes of getting the others to help with the corrections.
If you leave out enough, everything can become a beautiful story.
• Publication bias, file drawer problem
• Unspecified predictions, hidden unsustainable assumptions in theory.
Ideal vs Real
• Ideal scientists:
• Rational
• Unbiased
• Super intelligent
• Fair
• Communicates well
• Makes no mistakes
, Ideal
Real
Reviews are a lot more negative even when the only difference is negative results instead of positive
results.
• A letter from an editor of a major environmental/toxicological journal rejecting a manuscript
because of its negative findings included the statement:
o Unfortunately, we are not able to publish this manuscripts. The manuscript is very
well written and the study was well documented. Unfortunately, the negative
results translates into a minimal contribution to the field. We encourage you to
continue your work in this area and we will be glad to consider additional
manuscripts that you may prepare in the future.
