EXPERIMENTAL RESEARCH
STAT VIDEO 1: Basic of why we test
p-value
= the chance that you find your data (or a more extreme form), if you assume the null-hypothesis is true.
(if you assume there is no effect how likely are you to find this data)
≠ the chance that our hypothesis is true
≠ the chance that the null-hypothesis (i.e., there is no difference) is false
What does marginally significant mean (e.g., p = 0.071)
= Means the chance that you find this data if there was no effect is less likely (meaning the data is a bit
less surprising if you assume the null-hypothesis is true)
≠ does NOT mean there is a small effect. Can have large effect with marginally significant data.
≠ Does NOT mean that if you had more participant, it would have been significant
Standard deviation
𝑚𝑒𝑎𝑛(𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒)𝑜𝑓 𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑛𝑡𝑠−𝑎𝑣𝑒𝑟𝑎𝑔𝑒 𝑜𝑓 𝑔𝑟𝑜𝑢𝑝
SD =√
𝑡𝑜𝑡𝑎𝑙 𝑠𝑎𝑚𝑝𝑙𝑒 𝑠𝑖𝑧𝑒−1
Example
Grades in my class
Student 1 6
Student 2 7
Student 3 8
Student 4 5
Student 5 9
Mean = 7
((𝟔−𝟕)𝟐 +(𝟕−𝟕)𝟐 +(𝟖−𝟕)𝟐 +(𝟓−𝟕)𝟐 +(𝟗−𝟕)𝟐 )
SD = √ = 1.58
𝟓−𝟏
So see if everyone had a 6 or a 7 the SD would be smaller. So basically SD is the average deviation from the
mean so how does the average deviation from the mean (not exactly but for intuition purposes can say like
this).
Imagine hypothetical case that people in the U.S have the average length of the world population.
What happens to the mean and SD if we add all other countries to the sample?
1) The mean will stay almost the same (the U.S. was the average already)
2) The standard deviation will increase, as more shorter people (e.g., form Indonesia) and more taller
people (e.g., from the Netherlands) are added
Imagine that we had sampled 100 U.S. males and found their average length and the SD.
What happens if we sample 100 more participants?
1) The mean will become more precise estimate of the true length
2) The standard deviation will become more precise (but not necessarily smaller) estimates of the true
SD in the U.S. (SD will always be there and is unaffected of sample size whether it will be larger or
smaller)
1
,STAT VIDEO 2: The t-test (the independent samples one)
T-test: the base
If we observe a difference on a measure between two groups what makes us more likely to believe that
there actually a difference?
- A larger difference between the group These latter two help to determine the precision of the
- Less variation between individuals in each group estimate. Less variation and more observation can
- A larger sample size more accurately measure the mean of the group
T-test: formula
In other words all, else being equal:
- A larger difference between means give a larger t-value
- A smaller variance gives a larger t-value
- A larger sample size gives a larger t-value
We have a t-value. Now what?
We need to compare what we found (the t-test) to what we would expect if there is no actual difference.
This latter can (has been simulated). So most of the time if there is no effect you find a t-value of 0.
Then we compare the t-test and t-value that we found to
the simulated set. Stats program will do this for us.
This has a 0.013 chance we would find this data if there
was really no effect.
Effect size in t-test
For the effect size, we want to know how much of the variation we can explain.
What is variation? Standard Deviation*
How much can we explain? Well, the difference between means
*pooled version is weighted average
The difference between those 2 means is what we can explain by our created manipulation.
The formula is the difference between means, divided by the standard deviation. So in other words, reflects
how many standard deviations the means of two groups differ from each other.
2
, So how to report a t-test?
Participants who received a lottery ticket of which they could see the number of anticipated more regret
over exchanging it (M = 4.17, SD = 1.91) compared to participants who could not see the lottery number
because it was still in a sealed envelope (M = 3.56, SD = 1.94, t(288) = 2.41, p = 0.017, d = 0.32)
What does Cohen’ s d mean? How to interpret?
Idea (Cohen, 1988)
- 0.20 small
- 0.50 medium
- 0.80 large
→ I typically don’t like interpreting it his way with cut-off points. If you just give the effect size, you can
let the reader determine whether they think it is important enough. Small effects can still be influence if
they impact important behavior.
Let’s compare the t-test formula to Cohen’s d
The difference between the two groups does not change when group becomes larges, so effect size does
not change when changing the group size we only become more sure.
Assumptions in ‘’normal’’ t-test (& ANOVA)
1. Data is random subset of population (between and within groups)
2. DV at least interval level, but ordinal is also fine typically
3. Data is normally distributed
4. Variance in each condition is roughly equal
➔ 1 is important, else you do not know what cause the effect
- If your sample size is large and cells sizes are about equal, no worries about deviations from normal
distributions (especially if you use Welch T-test)
- Just make sure you inspect your data: if data is highly skewed, or with many outliers, etc, perhaps
use other analyses (non-parametric) or data transformations might be best.
3
STAT VIDEO 1: Basic of why we test
p-value
= the chance that you find your data (or a more extreme form), if you assume the null-hypothesis is true.
(if you assume there is no effect how likely are you to find this data)
≠ the chance that our hypothesis is true
≠ the chance that the null-hypothesis (i.e., there is no difference) is false
What does marginally significant mean (e.g., p = 0.071)
= Means the chance that you find this data if there was no effect is less likely (meaning the data is a bit
less surprising if you assume the null-hypothesis is true)
≠ does NOT mean there is a small effect. Can have large effect with marginally significant data.
≠ Does NOT mean that if you had more participant, it would have been significant
Standard deviation
𝑚𝑒𝑎𝑛(𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒)𝑜𝑓 𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑛𝑡𝑠−𝑎𝑣𝑒𝑟𝑎𝑔𝑒 𝑜𝑓 𝑔𝑟𝑜𝑢𝑝
SD =√
𝑡𝑜𝑡𝑎𝑙 𝑠𝑎𝑚𝑝𝑙𝑒 𝑠𝑖𝑧𝑒−1
Example
Grades in my class
Student 1 6
Student 2 7
Student 3 8
Student 4 5
Student 5 9
Mean = 7
((𝟔−𝟕)𝟐 +(𝟕−𝟕)𝟐 +(𝟖−𝟕)𝟐 +(𝟓−𝟕)𝟐 +(𝟗−𝟕)𝟐 )
SD = √ = 1.58
𝟓−𝟏
So see if everyone had a 6 or a 7 the SD would be smaller. So basically SD is the average deviation from the
mean so how does the average deviation from the mean (not exactly but for intuition purposes can say like
this).
Imagine hypothetical case that people in the U.S have the average length of the world population.
What happens to the mean and SD if we add all other countries to the sample?
1) The mean will stay almost the same (the U.S. was the average already)
2) The standard deviation will increase, as more shorter people (e.g., form Indonesia) and more taller
people (e.g., from the Netherlands) are added
Imagine that we had sampled 100 U.S. males and found their average length and the SD.
What happens if we sample 100 more participants?
1) The mean will become more precise estimate of the true length
2) The standard deviation will become more precise (but not necessarily smaller) estimates of the true
SD in the U.S. (SD will always be there and is unaffected of sample size whether it will be larger or
smaller)
1
,STAT VIDEO 2: The t-test (the independent samples one)
T-test: the base
If we observe a difference on a measure between two groups what makes us more likely to believe that
there actually a difference?
- A larger difference between the group These latter two help to determine the precision of the
- Less variation between individuals in each group estimate. Less variation and more observation can
- A larger sample size more accurately measure the mean of the group
T-test: formula
In other words all, else being equal:
- A larger difference between means give a larger t-value
- A smaller variance gives a larger t-value
- A larger sample size gives a larger t-value
We have a t-value. Now what?
We need to compare what we found (the t-test) to what we would expect if there is no actual difference.
This latter can (has been simulated). So most of the time if there is no effect you find a t-value of 0.
Then we compare the t-test and t-value that we found to
the simulated set. Stats program will do this for us.
This has a 0.013 chance we would find this data if there
was really no effect.
Effect size in t-test
For the effect size, we want to know how much of the variation we can explain.
What is variation? Standard Deviation*
How much can we explain? Well, the difference between means
*pooled version is weighted average
The difference between those 2 means is what we can explain by our created manipulation.
The formula is the difference between means, divided by the standard deviation. So in other words, reflects
how many standard deviations the means of two groups differ from each other.
2
, So how to report a t-test?
Participants who received a lottery ticket of which they could see the number of anticipated more regret
over exchanging it (M = 4.17, SD = 1.91) compared to participants who could not see the lottery number
because it was still in a sealed envelope (M = 3.56, SD = 1.94, t(288) = 2.41, p = 0.017, d = 0.32)
What does Cohen’ s d mean? How to interpret?
Idea (Cohen, 1988)
- 0.20 small
- 0.50 medium
- 0.80 large
→ I typically don’t like interpreting it his way with cut-off points. If you just give the effect size, you can
let the reader determine whether they think it is important enough. Small effects can still be influence if
they impact important behavior.
Let’s compare the t-test formula to Cohen’s d
The difference between the two groups does not change when group becomes larges, so effect size does
not change when changing the group size we only become more sure.
Assumptions in ‘’normal’’ t-test (& ANOVA)
1. Data is random subset of population (between and within groups)
2. DV at least interval level, but ordinal is also fine typically
3. Data is normally distributed
4. Variance in each condition is roughly equal
➔ 1 is important, else you do not know what cause the effect
- If your sample size is large and cells sizes are about equal, no worries about deviations from normal
distributions (especially if you use Welch T-test)
- Just make sure you inspect your data: if data is highly skewed, or with many outliers, etc, perhaps
use other analyses (non-parametric) or data transformations might be best.
3
