Chapter 3: The Phoneix of Statistics
Misconceptions about statistically significant results
- Misconception 1: A significant result means that the effect is important
- Very small and unimportant effects can be statistically significant with large
amounts of data while important and large effects may not be stat sig with small
amounts of data
- Misconception 2: A non-significant result means that the null hypothesis is true
- Non-sig result allows us to reject the alternative hypothesis but doesn’t tell us
that the effect size is 0
- Misconception 3: A significant result means that the null hypothesis is false
- Null hypothesis represents a statement of probability such as ‘if the null
hypothesis is correct, then this test statistic is highly unlikely’
Why does the NHST fail?
- It tries to test which of the null and alternative hypotheses are correct, but the
significance of the test provides no evidence about either hypothesis
- It encourages all or nothing thinking where if p<.05 then an effect is significant but if
p>.05 it is not
- NHST is influenced by the intentions of the scientist
- Empirical probability (long-run probability) is the proportion of events that have
the outcome in which you’re interested in an indefinitely large collective of events
- NHST is based on long-run probabilities with the alpha set at .05 probability of a
type 1 error so that across repeated identical experiments the probability of
making a type 1 error is 0.05; type 2 error is also a long run probaility
- For both the probability is not what is probable for every individual but for
what happens in the long run
- The p-value is the probability of getting a test statistic at least as large as the one
observed relative to all possible values of t(null) from an infinite number of
identical replications of the experiment
- Two different intentions for collecting data: collectin a certain number of data
points and collecting data up until a certain time point
- These influence how the p-values
Publication Bias
- Current incentive structure in science in invidiaulistic rather than collective where
individuals are rewarded for publishing and getting funding
- Publication bias: significant findings are about 7x more likely to be published
- Reviewers and editors tend to reject non-significnat results
- Researchers may selectively report their results to focus on sig findings
- Researcher degrees of freedom: researcher has many decisions to make when
designing and analysisng and they can be misused to exclude cases that make the
results non-signfincat
P-hacking and harking
, - P-hacking: researcher degrees of freedoms that lead to the selective reporting of
significant p-values
- Harking: hypothesizing after the results are known
- Both p-hacking and harking you are not controlling the type 1 error rate
- P-curve: number of p-values you would expect to get for each value of p
- Smaller p-values are more frequently reported than larger non-significant ones
EMBERS
- Effect Sizes
- Meta-analysis
- Bayesian Estimation
- Registration
- Sense
Sense
- 6 principles for using NHST
- You can use the value of p (not whether it is above or below an arbitrary
threshold) to indicate how incompatible the data are with the null hypothesis
- Smaller ps indicate more incompatibility with the null
- Don’t interpret p-values as probability that the hypothesis is true and not that data
was by random chance alone
- Decisions should not be based on whether p-value passes a scientific threshold
- Don’t p-hack: be transparent about everything
- Don’t confuse stat sign with practical importance (p-value doesn’t measure size
of the effect)
- Alone p-value doesnt provide a good measure of evidence regarding model or
hypothesis
Preregister
- Pre-registration: practice of making all aspects of your research process publicly
available before data collection
- Open science: movement to make the process, data and outcomes of research freely
available to everyone
Effect sizes:
- Problem with NHST is doesn’t tell us the importance of an effect
- Effect size: objective and standardized measure of the magnitude of observed effect
- Not affected by sample size (unlike p-values) , which makes it less misleading
than relying on p-values
- Common measures of effect size: cohen’s d, pearson’s correlation coefficient r, and
odds ratio
- When group sizes vary better to use cohen’s d because r can be biased
- Cohen’s d:
- Large effect: .8, medium .5 and small .2
Misconceptions about statistically significant results
- Misconception 1: A significant result means that the effect is important
- Very small and unimportant effects can be statistically significant with large
amounts of data while important and large effects may not be stat sig with small
amounts of data
- Misconception 2: A non-significant result means that the null hypothesis is true
- Non-sig result allows us to reject the alternative hypothesis but doesn’t tell us
that the effect size is 0
- Misconception 3: A significant result means that the null hypothesis is false
- Null hypothesis represents a statement of probability such as ‘if the null
hypothesis is correct, then this test statistic is highly unlikely’
Why does the NHST fail?
- It tries to test which of the null and alternative hypotheses are correct, but the
significance of the test provides no evidence about either hypothesis
- It encourages all or nothing thinking where if p<.05 then an effect is significant but if
p>.05 it is not
- NHST is influenced by the intentions of the scientist
- Empirical probability (long-run probability) is the proportion of events that have
the outcome in which you’re interested in an indefinitely large collective of events
- NHST is based on long-run probabilities with the alpha set at .05 probability of a
type 1 error so that across repeated identical experiments the probability of
making a type 1 error is 0.05; type 2 error is also a long run probaility
- For both the probability is not what is probable for every individual but for
what happens in the long run
- The p-value is the probability of getting a test statistic at least as large as the one
observed relative to all possible values of t(null) from an infinite number of
identical replications of the experiment
- Two different intentions for collecting data: collectin a certain number of data
points and collecting data up until a certain time point
- These influence how the p-values
Publication Bias
- Current incentive structure in science in invidiaulistic rather than collective where
individuals are rewarded for publishing and getting funding
- Publication bias: significant findings are about 7x more likely to be published
- Reviewers and editors tend to reject non-significnat results
- Researchers may selectively report their results to focus on sig findings
- Researcher degrees of freedom: researcher has many decisions to make when
designing and analysisng and they can be misused to exclude cases that make the
results non-signfincat
P-hacking and harking
, - P-hacking: researcher degrees of freedoms that lead to the selective reporting of
significant p-values
- Harking: hypothesizing after the results are known
- Both p-hacking and harking you are not controlling the type 1 error rate
- P-curve: number of p-values you would expect to get for each value of p
- Smaller p-values are more frequently reported than larger non-significant ones
EMBERS
- Effect Sizes
- Meta-analysis
- Bayesian Estimation
- Registration
- Sense
Sense
- 6 principles for using NHST
- You can use the value of p (not whether it is above or below an arbitrary
threshold) to indicate how incompatible the data are with the null hypothesis
- Smaller ps indicate more incompatibility with the null
- Don’t interpret p-values as probability that the hypothesis is true and not that data
was by random chance alone
- Decisions should not be based on whether p-value passes a scientific threshold
- Don’t p-hack: be transparent about everything
- Don’t confuse stat sign with practical importance (p-value doesn’t measure size
of the effect)
- Alone p-value doesnt provide a good measure of evidence regarding model or
hypothesis
Preregister
- Pre-registration: practice of making all aspects of your research process publicly
available before data collection
- Open science: movement to make the process, data and outcomes of research freely
available to everyone
Effect sizes:
- Problem with NHST is doesn’t tell us the importance of an effect
- Effect size: objective and standardized measure of the magnitude of observed effect
- Not affected by sample size (unlike p-values) , which makes it less misleading
than relying on p-values
- Common measures of effect size: cohen’s d, pearson’s correlation coefficient r, and
odds ratio
- When group sizes vary better to use cohen’s d because r can be biased
- Cohen’s d:
- Large effect: .8, medium .5 and small .2
