Lecture 1: Introduction, General Topics, Statistical Power
Parsimony =
How simple a theory is. How many assumptions a theory needs to make predictions → the
smaller the number, the better
Types of validity
1. internal validity = did the intervention rather than a confounded variable cause the
results?
2. external validity = how far can the results be generalized?
3. construct validity = which aspect of the intervention caused the results?
4. statistical validity = are the statistical conclusions correct?
Correlational research → How can I predict one variable if I know the other (regression)
If causality, then correlation BUT NOT if correlation, then causality
→ this is also true for temporal order;
if a causes b, a is before b
but not
if a is before b, a causes b
Floor effects =
When all your participants score low (using a clinical questionnaire for depression on
students)
→ Ceiling effects is when everyone scores high
Low sample size → low statistical power
Effect size and statistical power
There is a negative correlation between effect size and sample size. So the bigger the sample,
the smaller the effect.
It is about problems in generalizing from the small experimental sample to the population
1
,Effect size =
How large is the difference/correlation/relationship?
- Cohen’s d is a measure of how far two groups are apart compared to the variation
within each group.
- 0.2 is small, 0.5 is medium, 0.8 is large
- small effects always ask for huge samples
- measures of effect size =
d, f (=d/2 → ANOVA), partial eta2 (percentage of explained variance → also
ANOVA), r (pearson correlation)
Statistical power =
What is the probability that this effect will be statistically significant in an experiment?
What affects power?
1. effect size = larger effects are easier to find
2. sample size = effects are easier to find with many participants
3. alpha error = increasing the alpha error reduces the beta error
Why are so many small studies with large effects published?
1. Random fluctuation of effects in samples:
smaller sample → more fluctuation
This causes a flatter bell curve because there are more extremes
2. Publication bias favoring significant effects
Only the studies with significant effects will be published
Published effects that were chance results are not possible to replicate
2
, To increase power you:
- increase sample size
- increase effect size (increase systematic variance, decrease error variance, and
consider within-subjects instead of between-subjects)
Don’t believe in the effects found in published under-powered studies, no matter whether the
reported effect is small or large.
3
Parsimony =
How simple a theory is. How many assumptions a theory needs to make predictions → the
smaller the number, the better
Types of validity
1. internal validity = did the intervention rather than a confounded variable cause the
results?
2. external validity = how far can the results be generalized?
3. construct validity = which aspect of the intervention caused the results?
4. statistical validity = are the statistical conclusions correct?
Correlational research → How can I predict one variable if I know the other (regression)
If causality, then correlation BUT NOT if correlation, then causality
→ this is also true for temporal order;
if a causes b, a is before b
but not
if a is before b, a causes b
Floor effects =
When all your participants score low (using a clinical questionnaire for depression on
students)
→ Ceiling effects is when everyone scores high
Low sample size → low statistical power
Effect size and statistical power
There is a negative correlation between effect size and sample size. So the bigger the sample,
the smaller the effect.
It is about problems in generalizing from the small experimental sample to the population
1
,Effect size =
How large is the difference/correlation/relationship?
- Cohen’s d is a measure of how far two groups are apart compared to the variation
within each group.
- 0.2 is small, 0.5 is medium, 0.8 is large
- small effects always ask for huge samples
- measures of effect size =
d, f (=d/2 → ANOVA), partial eta2 (percentage of explained variance → also
ANOVA), r (pearson correlation)
Statistical power =
What is the probability that this effect will be statistically significant in an experiment?
What affects power?
1. effect size = larger effects are easier to find
2. sample size = effects are easier to find with many participants
3. alpha error = increasing the alpha error reduces the beta error
Why are so many small studies with large effects published?
1. Random fluctuation of effects in samples:
smaller sample → more fluctuation
This causes a flatter bell curve because there are more extremes
2. Publication bias favoring significant effects
Only the studies with significant effects will be published
Published effects that were chance results are not possible to replicate
2
, To increase power you:
- increase sample size
- increase effect size (increase systematic variance, decrease error variance, and
consider within-subjects instead of between-subjects)
Don’t believe in the effects found in published under-powered studies, no matter whether the
reported effect is small or large.
3
