Lecture 12: Logic and quality of research designs (Experimental designs)
An experimental research can be a medical situation or experience or a survey for example.
During the first lecture of IPRES we participated (students) in a research:
We had this situation:
We chose (by computer) a random number between 0 and 100. The number
selected and assigned to you is X = ___.
1) Do you think the percentage of countries, among all those in the United
Nations, that are in Africa is higher or lower than X?
2) Give your best estimate of the percentage of countries, among all those in
the United Nations, that are in Africa.
In order to respond we were divided into 3 groups each with a different random number: 10,
65, nothing. Anchoring effect: average answer:
- of those assigned no random number: 27%
- of those assigned the ‘random number’ 10: 23%
- of those assigned the ‘random number’ 65: 33%
RESULTS: We can see a visible difference in the estimated average of the 3 groups, this
shows that there was an anchoring effect (=when people use some information given even if
the information may not be useful or is random to guide them).
Terminology: Cause-effect relationship (how one cause has a or does a determined outcome
or effect). X(Y).
,Experimental designs: Basic principles, experiments are a valuable tool for research
because through them we systematically test our claims (are they true?). We usually test by
comparison of the outcome.
We had 2 plants and we put seeds: We wanna see what happens
+ plant 1: + H20
+ plant 2: - H20
An experimental design has a Research strategy; systematic testing of causal claims.
All the experiment relies in a critical element: the Controlled intervention/treatment =
manipulation of independent variable (cause) of interest.
The Isolation of ‘effect’ = difference between treatment and control groups in outcome.
It is important to remember that we can only use experimental settings if the 2 groups
compared are the same (ex: same kind of plant). That makes them comparable, we also
have to control the interventions. This is why the Ceteris paribus is so important => all else
equal: control of environment (lab) and random assignment to treatment and control group.
In the experiment lead in the lecture we saw that there was an anchoring effect and that it
was not in a lab → In this case the Ceteris paribus was established thanks to random
assignment.
Random assignment:
This is made to ensure that the division of participants in 2 groups happens in a random way.
A random assignment to treatment and control group ensures comparability of treatment and
control group, minimizes confounding factors (= omitted variable bias).
Ex: in the room there were different students with different individual characteristics, those
characteristics are important to the outcome. (love law, love IR, love politics, from Africa…)
We have to ensure that the two groups have the same characteristics and spread them
equally between the groups. It is important to minimize confounding factors because they
could influence the experience. (make sure each group has participants with the same
characteristics equally).
=> If you cannot control all the characteristics you will have to do a Random assignment:
And it can help prevent reactivity: fact that the participants react to the fact that they are
observed. (if participants don’t know the group they are in – blind).
, Also, it can help to prevent Rosenthal effect: when not even the researcher knows in which
group the participants are, this is make to avoid expectations about the outcome that could
influence the experience (if researchers and participants do not know who is in which group -
double blind).
=> Which comparisons make sense in experiments?
This Is important to check because systemic testing of causal explanations in experiments is
achieved through comparisons.
1) Post-test only:
Ex: In class we divide into 2 groups, one receives a treatment
(info with random number): Treatment group.
The other didn't receive anything: Control group. After this we
can compare the average of the answers of the 2 groups and
check whether there is a difference in how they answer,
Possible comparison: A – of post-test results in both groups. Is the
intervention effective?
BUT: Was randomization effective? - we can assume that but not
prove.
BUT: How does the change for individuals, on average, look like? -
has the treatment changed the answer of the group?
To check this we can made another type of test the Pre/Post and it
improves research design:
2) Pre-test/post-test two Groups design:
It adds a test before the treatment and adds other possible comparisons (in addition to A):
- B – of pre-tests in both groups. Did randomization work well?
- C – of pre-test/post-test in both groups. Does treatment group change over time (C)?
Are there confounding factors at play, e.g. changing environments (C1)?
BUT: Could pre-test have affected the post-test? - we can check this with the Solomon four
groups design.
, 3) Solomon Four Groups Design
Possible comparisons (in addition to A-C):
- D - of posttests between groups 1&2 and 3&4; if A and D differ, pretesting has
possibly affected the outcome.
- E –of pretest in group 2 and posttest in group 4: if there is a difference, a external
distortion may have caused the effect over time; causality?
- F (G) – of posttests between groups 1&3 (2&4) to see whether pre-testing has
affected the outcome.
Strengths and weaknesses of experiments
What is the biggest threat to the anchoring effect experiment?
a) that it did not take place in a lab
b) that the sample was not representative (only students)
c) that some students did cheat
d) that no two students are the same
All those options are serious and some refer to external and internal validity, internal validity
is usually more of a concern but trade off exist between both.
An experimental research can be a medical situation or experience or a survey for example.
During the first lecture of IPRES we participated (students) in a research:
We had this situation:
We chose (by computer) a random number between 0 and 100. The number
selected and assigned to you is X = ___.
1) Do you think the percentage of countries, among all those in the United
Nations, that are in Africa is higher or lower than X?
2) Give your best estimate of the percentage of countries, among all those in
the United Nations, that are in Africa.
In order to respond we were divided into 3 groups each with a different random number: 10,
65, nothing. Anchoring effect: average answer:
- of those assigned no random number: 27%
- of those assigned the ‘random number’ 10: 23%
- of those assigned the ‘random number’ 65: 33%
RESULTS: We can see a visible difference in the estimated average of the 3 groups, this
shows that there was an anchoring effect (=when people use some information given even if
the information may not be useful or is random to guide them).
Terminology: Cause-effect relationship (how one cause has a or does a determined outcome
or effect). X(Y).
,Experimental designs: Basic principles, experiments are a valuable tool for research
because through them we systematically test our claims (are they true?). We usually test by
comparison of the outcome.
We had 2 plants and we put seeds: We wanna see what happens
+ plant 1: + H20
+ plant 2: - H20
An experimental design has a Research strategy; systematic testing of causal claims.
All the experiment relies in a critical element: the Controlled intervention/treatment =
manipulation of independent variable (cause) of interest.
The Isolation of ‘effect’ = difference between treatment and control groups in outcome.
It is important to remember that we can only use experimental settings if the 2 groups
compared are the same (ex: same kind of plant). That makes them comparable, we also
have to control the interventions. This is why the Ceteris paribus is so important => all else
equal: control of environment (lab) and random assignment to treatment and control group.
In the experiment lead in the lecture we saw that there was an anchoring effect and that it
was not in a lab → In this case the Ceteris paribus was established thanks to random
assignment.
Random assignment:
This is made to ensure that the division of participants in 2 groups happens in a random way.
A random assignment to treatment and control group ensures comparability of treatment and
control group, minimizes confounding factors (= omitted variable bias).
Ex: in the room there were different students with different individual characteristics, those
characteristics are important to the outcome. (love law, love IR, love politics, from Africa…)
We have to ensure that the two groups have the same characteristics and spread them
equally between the groups. It is important to minimize confounding factors because they
could influence the experience. (make sure each group has participants with the same
characteristics equally).
=> If you cannot control all the characteristics you will have to do a Random assignment:
And it can help prevent reactivity: fact that the participants react to the fact that they are
observed. (if participants don’t know the group they are in – blind).
, Also, it can help to prevent Rosenthal effect: when not even the researcher knows in which
group the participants are, this is make to avoid expectations about the outcome that could
influence the experience (if researchers and participants do not know who is in which group -
double blind).
=> Which comparisons make sense in experiments?
This Is important to check because systemic testing of causal explanations in experiments is
achieved through comparisons.
1) Post-test only:
Ex: In class we divide into 2 groups, one receives a treatment
(info with random number): Treatment group.
The other didn't receive anything: Control group. After this we
can compare the average of the answers of the 2 groups and
check whether there is a difference in how they answer,
Possible comparison: A – of post-test results in both groups. Is the
intervention effective?
BUT: Was randomization effective? - we can assume that but not
prove.
BUT: How does the change for individuals, on average, look like? -
has the treatment changed the answer of the group?
To check this we can made another type of test the Pre/Post and it
improves research design:
2) Pre-test/post-test two Groups design:
It adds a test before the treatment and adds other possible comparisons (in addition to A):
- B – of pre-tests in both groups. Did randomization work well?
- C – of pre-test/post-test in both groups. Does treatment group change over time (C)?
Are there confounding factors at play, e.g. changing environments (C1)?
BUT: Could pre-test have affected the post-test? - we can check this with the Solomon four
groups design.
, 3) Solomon Four Groups Design
Possible comparisons (in addition to A-C):
- D - of posttests between groups 1&2 and 3&4; if A and D differ, pretesting has
possibly affected the outcome.
- E –of pretest in group 2 and posttest in group 4: if there is a difference, a external
distortion may have caused the effect over time; causality?
- F (G) – of posttests between groups 1&3 (2&4) to see whether pre-testing has
affected the outcome.
Strengths and weaknesses of experiments
What is the biggest threat to the anchoring effect experiment?
a) that it did not take place in a lab
b) that the sample was not representative (only students)
c) that some students did cheat
d) that no two students are the same
All those options are serious and some refer to external and internal validity, internal validity
is usually more of a concern but trade off exist between both.
