Research Workshop: Experiment
Week 1 – Tutorial 1
4 Elements of a True Experiment
1. Manipulation
a. The researcher manipulates one variable by changing its value to create a set of
two or more treatment conditions
b. Independent Variable
2. Measurement
a. The second variable is measured for a group of participants to obtain a set of
scores in each treatment condition
b. Dependent variable
3. Comparison
a. The scores in one treatment condition are compared to the scores in another
treatment condition
b. You need both conditions and measurements to do this
c. You compare e.g., Ad recognition between two conditions
4. Control
a. To measure the effect of the IV on the DV, all other variables are controlled to be
sure that they do not influence the variables being examined
b. The more control the better you can measure the causality between the
variables
o E.g., controlling for age, familiarity, gender
- There is a manipulation of a variable to create two conditions, after being exposed to this
manipulation a second variable is measured, this it compared between the condition to
look for a significant difference, while controlling for several other factors and having
randomization and the same procedure.
- E.g., Milgram Experiment
o Goal to see how far the participants would go to listen to authority
o Obedience to authority
o Experiment and students were part of the study
o Teacher paired with a student
o Teacher had to give shocks when they give a wrong answer
o Experimenter encourages the teacher to keep providing shocks
o The shock gets progressively more intense
o 65% of participants continued to the highest level of shock
o Milgram replicated this study with different demographics
- Is the Milgram experiment a true experiment...? No!
o Manipulation?
▪ No. Every treatment was the same, all participants had the same
treatment
o Measurement?
▪ Yes. Measurement was the severity of shocks, thus obedience to
authority, measurement of behaviour
o Comparison?
▪ No. No manipulation and thus no groups to compare
o Control?
,Research Workshop: Experiment
▪ No, because there is no manipulation thus, we can’t say that its causal,
but there were some attempts at control, e.g., same procedure and
reactions, script.
The Marshmallow Experiments
- The four elements of experimental design?
o Manipulation
▪ Study 1: No, because they gave all the kids the same option
▪ Study 2: Yes, all the kids had the same marshmallow test but they were
also split into the unreliable and reliable condition
o Measurement
▪ Yes. Both measured how long it took for the child to eat the marshmallow
o Comparison
▪ Study 1: No, because they were not split into conditions
▪ Study 2: Yes because they compared the length of time between kids in
each condition
o Control
▪ No. I don’t think there was much control, it wasn’t clear if the
experimenters used a script in either, or was randomized, but they were
almost the same.
▪ The second study did control for age because they were concerned with
that age group
▪ They could have controlled for gender, hungriness prior, tone
- Whether this is a true experiment or not and why?
o No, because neither fulfilled all 4 of the elements of a true experiment. If the
second study had more control variables, then it may be correct.
- What are the main differences between the two studies?
o The second study had manipulations.
Study 1:
- Child in room, given by an adult the option of waiting for another marshmallow while the
adult leaves the room, or eating the one in front of them.
Study 2:
- Looking into impulsivity of kids
- Kids were told that they can either use the ordinary art set or wait for the better art set
- Assigned kids to the reliable and the unreliable condition
o Reliable = the experimenter brought back a better art set
o Unreliable = the experimenter brought nothing back and thus the child had to
use the ordinary set.
- Then the same marshmallow test was performed.
- They found that the children waited longer for the marshmallow in the reliable condition
(12 minutes) than in the unreliable condition (3 minutes)
o Behaving impulsively might maximise the reward for those in the unreliable
condition
Correlation, causation, and extraneous variables
- Why can we not assume a causal relationship from a correlation?
,Research Workshop: Experiment
o You need to know why and how the causality is formed
o It doesn’t show the order of effects
o There may be third relationships that explain the relationship
- What do we need to prove causality? An Experimental research strategy
o The goal of the experimental research strategy is to establish the existence of a
cause-and-effect relationship between two variables
- IV = disclosure. Treatment = no disclose, disclosure. DV = attitude
- By manipulating the IV, we create different groups, and the groups receive different
treatments.
- The different scores of the DV, are caused by the IV treatments, unless there are
extraneous variables.
o = all variables beyond the IV and DV in the study. It could be anything! E.g.,
personality, IQ, gender, when the test is taken
o Problematic when they are confounding variables
o The goal of proper experimental design is to control these extraneous variables
and prevent them from becoming confounding variables
- Extraneous variables become confounding variables if…
o … it influences the DV. Something totally unrelated to the DV is not a threat
o … it varies systematically with the IV. A variable that changes randomly with not
relation to the IV is not a threat.
- Controlling extraneous variables → once a set of specific variables have been identified,
its possible to control for those.
- 3 methods to control for extraneous variables
1. Randomization
▪ = using an unbiased procedure to distribute participant across
conditions. Equal chance of participants ending up in different
treatments.
▪ Make sure that the differences between the groups are as small as
possible e.g., distribution of gender is equal over conditions
▪ Important that the sample is large enough to prevent individual
differences
2. Holding a variable constant
▪ Can be eliminated completely this way
▪ E.g., only selecting 30-year-old males for the study, you hold age and
gender constant
▪ Good for internal validity but you can’t generalise the results to other
demographics
3. Matching
▪ Match the levels of the variable across different treatment conditions
▪ Systematic division of participants. Pairs similar participants based on
age/gender, then assign them to different conditions
▪ The division is similar
Think about:
- The difference between correlation and causality (which was already in MCRS) and why
we can test causal relationships with an experiment but not with a survey.
- What's the difference?
- What element(s) of a true experiment concern testing causal relationships?
, Research Workshop: Experiment
Relevant Notes from MCRS (compulsory year 1 course)
o To prove causality (x->y)
1. Time order (x precedes y)
2. Meaningful covariance (x changes, y changes)
3. Non-spuriousness (only x influences y) > exclusive
True experiments
- Are experimental research designs that maximise internal validity and enable us to test
for causality
- Characteristics of true experiments:
o Manipulation → to ensure the cause proceeds the effect
o Comparison → to ensure the effect didn’t occur naturally
o Random assignment/randomization → ensures there are no other explanations
for the effect
- Manipulation
o When the researcher creates different level or conditions that represent different
values of the independent variable, while keeping the external variables constant
o Manipulation check = used to assess how effective a manipulation is
o Individual difference variables cannot be manipulated
- Randomization
o Provides a way of eliminating all possible systematic differences between
participants in different conditions at once
o Randomization failure = when there is an uneven distribution
o Statistical nuisance = the experimental group is not equal to the control group
o Randomization check = measure relevant variables to see if conditions are the
same
- Elements of experimental designs
1. DV is manipulated into 2 groups (exp & con) + manipulation check
2. DV is measured in the 2 groups
3. Results are compared to see if they statistically differ
4. All other variables that can influence the IV & DV are controlled
2. Random assignment = controlling for individual differences in experimental conditions →
when you have a large sample the biases will be evenly distributed & cancel each other
out → NEEDED for experiments and prevents selection bias
o Random sampling → who joins the study (external validity)
o Random assignment → who goes into what conditions (internal validity)
Types of Experimental Designs
1. Between-subjects design
a. Different participants are tested for manipulated conditions of the IV
Based on b. Compare different participants across conditions
level of
control
c. E.g. independent samples t-test
2. Within-subjects design
a. Same participants are tested in all manipulations of the IV
b. Compare same participants across conditions
c. e.g. Paired samples t-test
Week 1 – Tutorial 1
4 Elements of a True Experiment
1. Manipulation
a. The researcher manipulates one variable by changing its value to create a set of
two or more treatment conditions
b. Independent Variable
2. Measurement
a. The second variable is measured for a group of participants to obtain a set of
scores in each treatment condition
b. Dependent variable
3. Comparison
a. The scores in one treatment condition are compared to the scores in another
treatment condition
b. You need both conditions and measurements to do this
c. You compare e.g., Ad recognition between two conditions
4. Control
a. To measure the effect of the IV on the DV, all other variables are controlled to be
sure that they do not influence the variables being examined
b. The more control the better you can measure the causality between the
variables
o E.g., controlling for age, familiarity, gender
- There is a manipulation of a variable to create two conditions, after being exposed to this
manipulation a second variable is measured, this it compared between the condition to
look for a significant difference, while controlling for several other factors and having
randomization and the same procedure.
- E.g., Milgram Experiment
o Goal to see how far the participants would go to listen to authority
o Obedience to authority
o Experiment and students were part of the study
o Teacher paired with a student
o Teacher had to give shocks when they give a wrong answer
o Experimenter encourages the teacher to keep providing shocks
o The shock gets progressively more intense
o 65% of participants continued to the highest level of shock
o Milgram replicated this study with different demographics
- Is the Milgram experiment a true experiment...? No!
o Manipulation?
▪ No. Every treatment was the same, all participants had the same
treatment
o Measurement?
▪ Yes. Measurement was the severity of shocks, thus obedience to
authority, measurement of behaviour
o Comparison?
▪ No. No manipulation and thus no groups to compare
o Control?
,Research Workshop: Experiment
▪ No, because there is no manipulation thus, we can’t say that its causal,
but there were some attempts at control, e.g., same procedure and
reactions, script.
The Marshmallow Experiments
- The four elements of experimental design?
o Manipulation
▪ Study 1: No, because they gave all the kids the same option
▪ Study 2: Yes, all the kids had the same marshmallow test but they were
also split into the unreliable and reliable condition
o Measurement
▪ Yes. Both measured how long it took for the child to eat the marshmallow
o Comparison
▪ Study 1: No, because they were not split into conditions
▪ Study 2: Yes because they compared the length of time between kids in
each condition
o Control
▪ No. I don’t think there was much control, it wasn’t clear if the
experimenters used a script in either, or was randomized, but they were
almost the same.
▪ The second study did control for age because they were concerned with
that age group
▪ They could have controlled for gender, hungriness prior, tone
- Whether this is a true experiment or not and why?
o No, because neither fulfilled all 4 of the elements of a true experiment. If the
second study had more control variables, then it may be correct.
- What are the main differences between the two studies?
o The second study had manipulations.
Study 1:
- Child in room, given by an adult the option of waiting for another marshmallow while the
adult leaves the room, or eating the one in front of them.
Study 2:
- Looking into impulsivity of kids
- Kids were told that they can either use the ordinary art set or wait for the better art set
- Assigned kids to the reliable and the unreliable condition
o Reliable = the experimenter brought back a better art set
o Unreliable = the experimenter brought nothing back and thus the child had to
use the ordinary set.
- Then the same marshmallow test was performed.
- They found that the children waited longer for the marshmallow in the reliable condition
(12 minutes) than in the unreliable condition (3 minutes)
o Behaving impulsively might maximise the reward for those in the unreliable
condition
Correlation, causation, and extraneous variables
- Why can we not assume a causal relationship from a correlation?
,Research Workshop: Experiment
o You need to know why and how the causality is formed
o It doesn’t show the order of effects
o There may be third relationships that explain the relationship
- What do we need to prove causality? An Experimental research strategy
o The goal of the experimental research strategy is to establish the existence of a
cause-and-effect relationship between two variables
- IV = disclosure. Treatment = no disclose, disclosure. DV = attitude
- By manipulating the IV, we create different groups, and the groups receive different
treatments.
- The different scores of the DV, are caused by the IV treatments, unless there are
extraneous variables.
o = all variables beyond the IV and DV in the study. It could be anything! E.g.,
personality, IQ, gender, when the test is taken
o Problematic when they are confounding variables
o The goal of proper experimental design is to control these extraneous variables
and prevent them from becoming confounding variables
- Extraneous variables become confounding variables if…
o … it influences the DV. Something totally unrelated to the DV is not a threat
o … it varies systematically with the IV. A variable that changes randomly with not
relation to the IV is not a threat.
- Controlling extraneous variables → once a set of specific variables have been identified,
its possible to control for those.
- 3 methods to control for extraneous variables
1. Randomization
▪ = using an unbiased procedure to distribute participant across
conditions. Equal chance of participants ending up in different
treatments.
▪ Make sure that the differences between the groups are as small as
possible e.g., distribution of gender is equal over conditions
▪ Important that the sample is large enough to prevent individual
differences
2. Holding a variable constant
▪ Can be eliminated completely this way
▪ E.g., only selecting 30-year-old males for the study, you hold age and
gender constant
▪ Good for internal validity but you can’t generalise the results to other
demographics
3. Matching
▪ Match the levels of the variable across different treatment conditions
▪ Systematic division of participants. Pairs similar participants based on
age/gender, then assign them to different conditions
▪ The division is similar
Think about:
- The difference between correlation and causality (which was already in MCRS) and why
we can test causal relationships with an experiment but not with a survey.
- What's the difference?
- What element(s) of a true experiment concern testing causal relationships?
, Research Workshop: Experiment
Relevant Notes from MCRS (compulsory year 1 course)
o To prove causality (x->y)
1. Time order (x precedes y)
2. Meaningful covariance (x changes, y changes)
3. Non-spuriousness (only x influences y) > exclusive
True experiments
- Are experimental research designs that maximise internal validity and enable us to test
for causality
- Characteristics of true experiments:
o Manipulation → to ensure the cause proceeds the effect
o Comparison → to ensure the effect didn’t occur naturally
o Random assignment/randomization → ensures there are no other explanations
for the effect
- Manipulation
o When the researcher creates different level or conditions that represent different
values of the independent variable, while keeping the external variables constant
o Manipulation check = used to assess how effective a manipulation is
o Individual difference variables cannot be manipulated
- Randomization
o Provides a way of eliminating all possible systematic differences between
participants in different conditions at once
o Randomization failure = when there is an uneven distribution
o Statistical nuisance = the experimental group is not equal to the control group
o Randomization check = measure relevant variables to see if conditions are the
same
- Elements of experimental designs
1. DV is manipulated into 2 groups (exp & con) + manipulation check
2. DV is measured in the 2 groups
3. Results are compared to see if they statistically differ
4. All other variables that can influence the IV & DV are controlled
2. Random assignment = controlling for individual differences in experimental conditions →
when you have a large sample the biases will be evenly distributed & cancel each other
out → NEEDED for experiments and prevents selection bias
o Random sampling → who joins the study (external validity)
o Random assignment → who goes into what conditions (internal validity)
Types of Experimental Designs
1. Between-subjects design
a. Different participants are tested for manipulated conditions of the IV
Based on b. Compare different participants across conditions
level of
control
c. E.g. independent samples t-test
2. Within-subjects design
a. Same participants are tested in all manipulations of the IV
b. Compare same participants across conditions
c. e.g. Paired samples t-test
