Applied Research Methods – Colleges
Week 1
General topics
Scientific research and theory
Types of research
- Observations: finding phenomena
- Correlations and quasi-experiments: finding relationships
between variables. By finding the correlation, we still don’t know
why there is a relation
- Experiments: finding causal explanations
- All of them: developing and testing theories of experience and
behaviour
How do you tell a good theory from a bad one?
- Precision.
- Parsimony: we try to explain a phenomenon with as few
assumptions as possible. The fewer assumptions you need to
predict something, the better
- Testability and falsifiability: if it can be tested, and it can be
proven wrong
The validity of scientific research
Types of validity
- Internal validity: did the intervention rather than a confounded
variable cause the results?
- External validity: how far can these results be generalized?
- Construct validity: Which aspects of the intervention caused the
results?
- Statistical validity: are the statistical conclusions correct?
Correlational research
Correlational research questions
- How closely are two variables related? correlation
- How can I predict one variable if I know the other regression
How can correlations be used and interpreted
- Correlation: can be used for direction and size
- Regression: can be used to predict, this can be more or less
precise depending on the correlation
Beware of causal interpretations
Correlation and causality
1
, An example of the causality problem
Depressed patients think more negatively about themselves than
others negative correlation of depression and thinking, but how
do they influence each other?
- Negative thinking causes depression?
- Depression causes negative thinking?
- Depression and negative thinking cause each other?
- A third variable (genetic, neurological) causes both
depression and negative thinking?
More examples of dubious causalities
- The number of crimes and the number of churches in a cite
are correlated does religion cause crime?
- Sales of ice cream and drowning rates are correlated
does ice cream cause drowning?
- Shoe size is positively correlated with alcoholism, and
negatively correlated with anxiety do big feet cause
alcoholism, but protect from anxiety?
The relation is not symmetric: if causality, then correlation. But not:
if correlation, then causality
And temporal order does not prove causality, either: If A is the
cause of B, A must happen before. But not: if A happens before B, A
is the cause of B
- Even if two variables are both correlated and temporally
ordered, the earlier one does not have to be the cause of the
later one
- Correlation is a necessary, but not a sufficient precondition for
causation
The one and only way to establish a causal relationship underlying a
correlation is to conduct an experiment
Variables in experiments
Independent variables (manipulated by experimenter)
- What is a good independent variable?
- How many levels of the variable? The more levels you have, the
better you can judge whether there is for example a linear
regression. But also for every level you add, it might force you to
add more participants
Dependent variables (measured by experimenter)
- What is a good dependent variable?
2
, - Beware of floor effects and ceiling effects. Try to avoid situations
where everybody scores low or high independent from what you
are measuring
Control variables (controlled by experimenter)
- Holding them constant. You don’t want to mix up your dependent
and independent variables
- Turning them into independent variables. If you don’t want to
control them or can’t control them, manipulate them
Between-subjects vs. within-subjects designs
Between-subjects designs (independent groups)
- Every subject experiences only one level of the independent
variable: random assignment to the treatment we want to
compare. Random assignment is very powerful, but it requires a
large number of participants
Within-subjects designs (repeated measures)
- Every subject experiences every of the independent variable:
order effects?. It is easier to find the difference between groups
with a within-subjects design than a between-subject design
because it is more powerful.
- You have to counterbalance the order because the results
may differ if you use easy – intermediate – hard vs. hard –
intermediate – easy conditions.
Problems of experimental designs
Particularly critical in clinical psychology
- Quasi-experiments instead of random assignment
- External validity
- Laboratory vs. everyday life. Is a laboratory setting
generalizable to everyday life?
- Patients vs. analogue populations. E.g., do highly
anxious student generalize to phobic patients?
- Low sample size low statistical power
Effect size and statistical power
Effect size and statistical power: why bother?
- How many participants will I probably need in my study?
- Why do so many experiments in psychology yield non-significant
results?
- Why should I better not believe many of the significant results I
read about?
3
, Two types of errors
Problems in generalizing from the small
experimental sample to the population
ß: false negative rate
α: false positive rate
Why are the effect size and statistical power there?
Effect size: how large is a difference/correlation/relationship?
Statistical power: what is the probability that this effect will be
statistically significant in an experiment?
Situations where this is important
- Experiment in preparation: determine necessary sample size
- Experiment completed: determine power of the experiment
- Evaluation of published studies: are the effects for real?
Effect sizes: Cohen’s d as a simple example
Situation: comparing two group means by a t-test
In a t-test you assume that you have two groups with
both a mean that differs from each other
Effect size d
How large is d typically in Psychology?
0.5 is considered a medium size effect. This
means that these two groups overlap
considerately. The study is average.
0.2 is considered small
0.8 is considered large
What affects power?
Effects size: larger effects are easier to find than small effects. You
need a huge sample to detect a small effect
Sample size: effects are easier to find with many participants.
Alpha error: increasing the alpha error reduces the
beta error. If you are more willing to accept false
positive, you will have fewer false negative and the
other way around.
4
Week 1
General topics
Scientific research and theory
Types of research
- Observations: finding phenomena
- Correlations and quasi-experiments: finding relationships
between variables. By finding the correlation, we still don’t know
why there is a relation
- Experiments: finding causal explanations
- All of them: developing and testing theories of experience and
behaviour
How do you tell a good theory from a bad one?
- Precision.
- Parsimony: we try to explain a phenomenon with as few
assumptions as possible. The fewer assumptions you need to
predict something, the better
- Testability and falsifiability: if it can be tested, and it can be
proven wrong
The validity of scientific research
Types of validity
- Internal validity: did the intervention rather than a confounded
variable cause the results?
- External validity: how far can these results be generalized?
- Construct validity: Which aspects of the intervention caused the
results?
- Statistical validity: are the statistical conclusions correct?
Correlational research
Correlational research questions
- How closely are two variables related? correlation
- How can I predict one variable if I know the other regression
How can correlations be used and interpreted
- Correlation: can be used for direction and size
- Regression: can be used to predict, this can be more or less
precise depending on the correlation
Beware of causal interpretations
Correlation and causality
1
, An example of the causality problem
Depressed patients think more negatively about themselves than
others negative correlation of depression and thinking, but how
do they influence each other?
- Negative thinking causes depression?
- Depression causes negative thinking?
- Depression and negative thinking cause each other?
- A third variable (genetic, neurological) causes both
depression and negative thinking?
More examples of dubious causalities
- The number of crimes and the number of churches in a cite
are correlated does religion cause crime?
- Sales of ice cream and drowning rates are correlated
does ice cream cause drowning?
- Shoe size is positively correlated with alcoholism, and
negatively correlated with anxiety do big feet cause
alcoholism, but protect from anxiety?
The relation is not symmetric: if causality, then correlation. But not:
if correlation, then causality
And temporal order does not prove causality, either: If A is the
cause of B, A must happen before. But not: if A happens before B, A
is the cause of B
- Even if two variables are both correlated and temporally
ordered, the earlier one does not have to be the cause of the
later one
- Correlation is a necessary, but not a sufficient precondition for
causation
The one and only way to establish a causal relationship underlying a
correlation is to conduct an experiment
Variables in experiments
Independent variables (manipulated by experimenter)
- What is a good independent variable?
- How many levels of the variable? The more levels you have, the
better you can judge whether there is for example a linear
regression. But also for every level you add, it might force you to
add more participants
Dependent variables (measured by experimenter)
- What is a good dependent variable?
2
, - Beware of floor effects and ceiling effects. Try to avoid situations
where everybody scores low or high independent from what you
are measuring
Control variables (controlled by experimenter)
- Holding them constant. You don’t want to mix up your dependent
and independent variables
- Turning them into independent variables. If you don’t want to
control them or can’t control them, manipulate them
Between-subjects vs. within-subjects designs
Between-subjects designs (independent groups)
- Every subject experiences only one level of the independent
variable: random assignment to the treatment we want to
compare. Random assignment is very powerful, but it requires a
large number of participants
Within-subjects designs (repeated measures)
- Every subject experiences every of the independent variable:
order effects?. It is easier to find the difference between groups
with a within-subjects design than a between-subject design
because it is more powerful.
- You have to counterbalance the order because the results
may differ if you use easy – intermediate – hard vs. hard –
intermediate – easy conditions.
Problems of experimental designs
Particularly critical in clinical psychology
- Quasi-experiments instead of random assignment
- External validity
- Laboratory vs. everyday life. Is a laboratory setting
generalizable to everyday life?
- Patients vs. analogue populations. E.g., do highly
anxious student generalize to phobic patients?
- Low sample size low statistical power
Effect size and statistical power
Effect size and statistical power: why bother?
- How many participants will I probably need in my study?
- Why do so many experiments in psychology yield non-significant
results?
- Why should I better not believe many of the significant results I
read about?
3
, Two types of errors
Problems in generalizing from the small
experimental sample to the population
ß: false negative rate
α: false positive rate
Why are the effect size and statistical power there?
Effect size: how large is a difference/correlation/relationship?
Statistical power: what is the probability that this effect will be
statistically significant in an experiment?
Situations where this is important
- Experiment in preparation: determine necessary sample size
- Experiment completed: determine power of the experiment
- Evaluation of published studies: are the effects for real?
Effect sizes: Cohen’s d as a simple example
Situation: comparing two group means by a t-test
In a t-test you assume that you have two groups with
both a mean that differs from each other
Effect size d
How large is d typically in Psychology?
0.5 is considered a medium size effect. This
means that these two groups overlap
considerately. The study is average.
0.2 is considered small
0.8 is considered large
What affects power?
Effects size: larger effects are easier to find than small effects. You
need a huge sample to detect a small effect
Sample size: effects are easier to find with many participants.
Alpha error: increasing the alpha error reduces the
beta error. If you are more willing to accept false
positive, you will have fewer false negative and the
other way around.
4
