Lecture 11 – read Heiman chapter on Canvas
Design of an experiment: how you setup an experiment. No statistical technique can fix a poorly
designed experiment.
Goal experiments: Determine whether independent variable (IV) has an effect on the dependent
variable (DV) – causal relationship: we try to achieve this goal with: 1. High external validity
2. High internal validity 3. High statistical power
1. External validity →the degree to which the conclusions of a study can be generalized to other
people, geographical location, context, time, and duration of effect
2. Internal validity →the degree to which the relationship between the IV and DV indeed (only)
reflects the relationship between IV and DV
3. Power →maximizing the probability of finding an effect between IV and DV, so making an effort to
decrease MSw
Sampling: Optimal situation →simple random sample. The larger
the simple random sample, the less likely it is that you have a
nonrepresentative sample →higher external validity of the study
A convenience sample rather than a simple random sample is
often used in psychology. Criticism on psychology as a science:
“psychology is not the study of humans, but it is the study of first
year psychology students”. Large threat to the external validity,
because sample is not really representative. And...
Volunteer bias →only those subjects are included that are willing to participated in a study. Subject
sophistication →subjects are knowledgeable about the topic of the research. And lower power if
variation of the DV is lower in a convenience sample than in population (restriction of range)
Sampling: Conclusions: If psychological processes are studied that are expected to be different in
different groups of people, a simple random sample is preferable to achieve the goal of an experiment.
If psychological processes do not differ between different groups of people, a convenience sample is
allowed. Psychologists use convenience samples extremely often, even when psychological processes
differ between different groups of people. Psychologists should use simple random samples more often!
Controlling for confounding variables: There is confounding if participants differ between conditions.
For example, IQ is a confounding variable if we conduct an experiment to study whether a new teaching
method improves students’ performance. Confounding can be caused by bad luck if random assignment
was used. Confounding variables decrease the internal validity and power
Confounding variables decrease internal validity. Suppose that the confounding variable is related to the
DV. Persons between levels of the IV differ on the confounding variable. Then there are also differences
in the DV between levels, not directly caused by differences in the IV. Therefore, problems with internal
validity → relationship is not IV causes DV, but confounding variable causing DV
Design of an experiment: how you setup an experiment. No statistical technique can fix a poorly
designed experiment.
Goal experiments: Determine whether independent variable (IV) has an effect on the dependent
variable (DV) – causal relationship: we try to achieve this goal with: 1. High external validity
2. High internal validity 3. High statistical power
1. External validity →the degree to which the conclusions of a study can be generalized to other
people, geographical location, context, time, and duration of effect
2. Internal validity →the degree to which the relationship between the IV and DV indeed (only)
reflects the relationship between IV and DV
3. Power →maximizing the probability of finding an effect between IV and DV, so making an effort to
decrease MSw
Sampling: Optimal situation →simple random sample. The larger
the simple random sample, the less likely it is that you have a
nonrepresentative sample →higher external validity of the study
A convenience sample rather than a simple random sample is
often used in psychology. Criticism on psychology as a science:
“psychology is not the study of humans, but it is the study of first
year psychology students”. Large threat to the external validity,
because sample is not really representative. And...
Volunteer bias →only those subjects are included that are willing to participated in a study. Subject
sophistication →subjects are knowledgeable about the topic of the research. And lower power if
variation of the DV is lower in a convenience sample than in population (restriction of range)
Sampling: Conclusions: If psychological processes are studied that are expected to be different in
different groups of people, a simple random sample is preferable to achieve the goal of an experiment.
If psychological processes do not differ between different groups of people, a convenience sample is
allowed. Psychologists use convenience samples extremely often, even when psychological processes
differ between different groups of people. Psychologists should use simple random samples more often!
Controlling for confounding variables: There is confounding if participants differ between conditions.
For example, IQ is a confounding variable if we conduct an experiment to study whether a new teaching
method improves students’ performance. Confounding can be caused by bad luck if random assignment
was used. Confounding variables decrease the internal validity and power
Confounding variables decrease internal validity. Suppose that the confounding variable is related to the
DV. Persons between levels of the IV differ on the confounding variable. Then there are also differences
in the DV between levels, not directly caused by differences in the IV. Therefore, problems with internal
validity → relationship is not IV causes DV, but confounding variable causing DV
