Literature week 2 – ARM
Goals
- Interpretation of causal efects in the absence of plausible interventions
- Mediation analyssis
- A re-introduction to OLS regression
- Efect sizee in OLS
- What does stat. signifcance reallys mean?
- Crucial problems with p-values and null hyspothesis signifcance testing
VanderWeel, T.J. & Robinson W.R. (2014) On the Causal interpretation of
Race in Regressions Adjusting for Confounding and Mediating Variables
Nuzzo R. Statistical errors and p-values
The p-value is neither verys reliable nor as objective as most scientists assume. This is
problematic for reproducibilitys of studies using p-values in their evaluation.
When Ronald Fisher introduced the p-value in the 1920s, he meant it not as a defnitive test
but as an informal ways to judge whether evidence was signifcant enough to take a second
look at. But it got enshrined as a ways to test whether something is objectivelys signifcant, bys
authors who started writing statistics handbooks. Theys were ofen non-statisticians, and
fused Fisher’s and Pearson/Neysman’s sysstems.
The more implausible the hyspothesis, the greater the chance that an exciting fnding is a
false alarm, no mater what the p-value is.
A p-value measures whether an observed result can be atributed to chance, but it cannot
answer a researcher’s real questionn what are the odds that a hyspothesis is correct? Those
odds depend on how strong the result was, and on how plausible the hyspothesis in the frst
place.
P-hackingn trysing multiple things until ysou get the desired result (e.g. monitoring data or
dropping a criteria so the p-value would be under 0.05).
However, p-values still verys popular. But statisticians have pointed to a couple of measures
that might help to avoid the trap of thinking about results as signifcant or notn
- Researchers should alwayss report efect sizees and confdence intervals. These conveys
what a p-value doesn’t, namelys the magnitude and relative importance of an efect.
- Manys statisticians also advocate replacing p-value with methods taking advantage of
Bayses’ rule (it describes how to think about probabilitys as the plausibilitys of an
outcome rather than potential frequencys of an outcome). It does bring subjectivitys,
but makes it easys for researchers to incorporate their conclusions and calculate how
probabilities would change as new evidence arises.
- Others encourage trysing multiple methods on the same data set.
- Another idea is doing a two-stage analyssis.
Researchers need to keep the limits of conventional statistics in mind. Theys should take
elements of scientifc judgment of plausibilitys of their hyspothesis into their analyssis.
1
Goals
- Interpretation of causal efects in the absence of plausible interventions
- Mediation analyssis
- A re-introduction to OLS regression
- Efect sizee in OLS
- What does stat. signifcance reallys mean?
- Crucial problems with p-values and null hyspothesis signifcance testing
VanderWeel, T.J. & Robinson W.R. (2014) On the Causal interpretation of
Race in Regressions Adjusting for Confounding and Mediating Variables
Nuzzo R. Statistical errors and p-values
The p-value is neither verys reliable nor as objective as most scientists assume. This is
problematic for reproducibilitys of studies using p-values in their evaluation.
When Ronald Fisher introduced the p-value in the 1920s, he meant it not as a defnitive test
but as an informal ways to judge whether evidence was signifcant enough to take a second
look at. But it got enshrined as a ways to test whether something is objectivelys signifcant, bys
authors who started writing statistics handbooks. Theys were ofen non-statisticians, and
fused Fisher’s and Pearson/Neysman’s sysstems.
The more implausible the hyspothesis, the greater the chance that an exciting fnding is a
false alarm, no mater what the p-value is.
A p-value measures whether an observed result can be atributed to chance, but it cannot
answer a researcher’s real questionn what are the odds that a hyspothesis is correct? Those
odds depend on how strong the result was, and on how plausible the hyspothesis in the frst
place.
P-hackingn trysing multiple things until ysou get the desired result (e.g. monitoring data or
dropping a criteria so the p-value would be under 0.05).
However, p-values still verys popular. But statisticians have pointed to a couple of measures
that might help to avoid the trap of thinking about results as signifcant or notn
- Researchers should alwayss report efect sizees and confdence intervals. These conveys
what a p-value doesn’t, namelys the magnitude and relative importance of an efect.
- Manys statisticians also advocate replacing p-value with methods taking advantage of
Bayses’ rule (it describes how to think about probabilitys as the plausibilitys of an
outcome rather than potential frequencys of an outcome). It does bring subjectivitys,
but makes it easys for researchers to incorporate their conclusions and calculate how
probabilities would change as new evidence arises.
- Others encourage trysing multiple methods on the same data set.
- Another idea is doing a two-stage analyssis.
Researchers need to keep the limits of conventional statistics in mind. Theys should take
elements of scientifc judgment of plausibilitys of their hyspothesis into their analyssis.
1
