Advanced Research Methods
Advanced Research Methods
Week 1 Causal inference
Lecture 1
Causal inference study
The effect of X on Y so the effect of something on something else.
Problems
Small sample (only 41 women where included in this study).
o Is this always a problem? No, it also has a relationship with the effect or with the
outcomes and the homogeneity in the outcomes.
Study performed or financed by commercial company.
o Is this always fatal? This is always a challenge, but you can do a few things to protect
your independence as a researcher.
No control group.
o This is indeed an essential omission.
o What without treatment?
o Potential regression towards the mean (would it also be better without the
intervention? We don’t know).
What do we want to know?
Not interested in the outcome per se: ‘how many imperfections’.
Interested in the role of treatment in achieving this outcome: ‘less imperfections than without
True Match Minerals’.
Conclusion: no meaningful causal conclusion can be drawn from this study.
Causation: towards a usable, formal framework
Formal definition (Hernàn/Robins): “in an individual, a treatment has a causal effect if the
outcome under treatment 1 would be different from the outcome under treatment 2”. Then you
can say:
o Wat would have happened?
o What will happen?
So in the case about True Match Minerals:
Women A uses True Match Minerals: 2 bad spots.
Women A does not use True Match Minerals: 5 bad spots.
Individual treatment effect: 3 spots (60%).
Average treatment effect: average of all individual effects in a population.
Potential outcomes | 1
Causal effect:
Y = outcome.
A = treatment.
1 = yes (received treatment).
0 = no (received no treatment).
I = individual.
Does not equal.
This says that the potential outcomes are different.
Potential outcomes | 2
1
, Advanced Research Methods
Let’s improve the experiment and add a control group.
The outcome of user K with Ya = 1 = 1 (improvement with
treatment).
The outcome of user K with Ya = 0 = 0 (no improvement
without treatment).
Treatment effect for K: 1 – 0 = 1.
Average treatment effect is the average of the last column.
Not all potential outcomes are observed
Counterfactual outcome: potential outcome that is not observed because the subject did not
experience the treatment (‘counter the fact’).
o It could have happened, but it didn’t. For user K we
only see the outcome with treatment. For user M and
N we only see the outcome without problem. This is
the fundamental problem of causal effects.
Potential outcome (Ya=1) is fractural for some objects, and
counterfactual for others.
Fundamental problem
Individual causal effect cannot be observed. Except under extremely strong (and generally
unreasonable) assumptions.
o You don’t know what would happened without the treatment.
Average causal effect cannot be inferred from individual estimates. Causal inference as a
missing data problem.
We need a different approach to causal effects.
Identifiability conditions
Average causal effects can still be determined under certain conditions.
‘Observing’ the counterfactual: what would have happened?
Based on population averages, causal effects can be estimated if three identifiability
conditions hold:
o Positivity.
o Consistency.
o Exchangeability.
If the conditions are met, then association of exposure and outcome is unbiased estimate of
causal effect.
Simple example
Go to the Lijnbaan in Rotterdam.
We ask everyone ‘are you carrying a cigarette lighter?’
Come back after twenty years: who is healthier?
Causal questions: what is the effect of carrying a lighter on health?
Positivity | 1
Observe ‘what would have happened if…’.
This is about the sample and the way it was composed.
‘Positive probability’ of being assigned to each of the treatment levels.
Units are assigned to all relevant ‘treatments’:
o People with and people without a cigarette lighter.
o People with lighter could also not have had a lighter, and vice verca.
o This was not the case in the example of L’oreal: 100% was assigned to True Match
Mineral, 0% to comparison group. Users could not not have used it.
Control group.
2
, Advanced Research Methods
Consistency
Observe ‘what would have happened if…’.
Define ‘if’: clear definition of ‘treatments’.
Hernàn: does water kill? What do you mean by water? You have to be very specific.
Is it consistent?
o How healthy is broccoli? But how much? How often? What else are you eating? What
are we comparing this to?
o Effect of obesity on health? Obesity is not informative. We have to know how people
did or did not have obesity.
o Effect of obesity on job prospects? In this question it doesn’t matter how people got
obeses. This is a consistent definition.
o Effect of healthcare spending on mortality? What is the money going to be spend on?
How would it be spend without? This isn’t a consistent definition.
o Carrying a cigarette lighter? This might be (in hand or in pocket).
Exchangeability | 1
Observe ‘what would have happened if…’.
Treatment groups are exchangeable: it does not matter who gets treatment A and who gets
treatment B.
Notation:
‘Potential outcomes are independent of the treatment that was actually received’.
Are people with and without lighters exchangeable (similar in other respects).
Exchangeability | 2
Observe ‘what would have happened if…’.
It may be necessary to take other factors into account (adjustment).
o Within smoking group, are people with and without lighter exchangeable?
o Within non-smoking group, are people with and without lighter exchangeable?
Association can be ascribed to treatment effect.
Positivity | 2
Units are assigned to all relevant ‘treatments’ within levels of adjustment factors.
‘Positive probability’ of being assigned to each of the treatment levels. We need:
o Smokers with cigarette lighter.
o Smokers without cigarette lighter.
o Non-smokers with cigarette lighter.
o Non-smokers without cigarette lighter.
Stratification
This is when you divide the sample in different groups (according to the value of one variable). In this
case the variable is smoking.
First without stratification: with lighter
65 people are healthy, without lighter 165
people are healthy. We don’t believe we
achieved exchangeability.
With stratification: 50% of smokers is
healthy with lighter and 50% without
lighter. There is within the smoking
group no association with health. Neither
with the non-smoking group.
We had consistency (we know what we mean by
carrying a ligher), we had positivity (enough observations in the different groups) and we achieved
exchangeability (if smoking is the only things that counts). If we believe all the condition are met (we
3
, Advanced Research Methods
cannot measure) then we have an unbiased estimate of the causal effect of carrying a lighter. In this
case the effect is 0 (there is no effect).
Meeting the conditions: Randomized Controlled Trial (RCT)
Select patients.
Randomly assign to the treatment groups.
o Random: exchangeability.
o Random: positivity.
o Consistency: you have to identify the interventions so you have to know what these
groups are.
RCTs versus observational studies
RCT
Limited generalizability (external validity) due to treatment protocol and patient selection.
o You treat people in very specific circumstances (controlled), what does this say about
the real-world?
Practical, ethical considerations.
Observational (non-randomised) study
Real-world outcomes (advantage).
Availability of data (can be collected or are available).
Internal validity threatened by lack of exchangeability.
Positivity and consistency need explicit attention.
Association does not equal causation
In many cases, we are interested in causal effects, not just associations.
Association: statistical relationship.
Causation: difference between potential outcomes.
This association equals this difference if identifiability conditions hold.
We need:
o Theory/subject knowledge.
o Causal structure.
And we design the analysis accordingly.
Adjustment to improve exchangeability
Small number of factors? Stratification is possible. Other ways:
o Matching.
o Weighting.
o Regression analysis.
Complete and correct adjustment leads to exchangeability.
So … what should you adjust for in your analysis?
Traditional selection strategies
Correlation matrix: select variables with sign. association with outcome.
Stepwise backward selection:
o Start with all variables in regression model.
o Remove the variable that is the least statistically significant.
o Repeat steps.
Or: retain variable if removal leads to substantial change in effect estimate (PC lab).
Adjust for confounders, which are defined as being:
o Associated with the exposure, and;
o Conditionally associated with the outcome, given the exposure;
o Not in the causal pathway between exposure and outcome.
These are alternatives, but don’t.
4
Advanced Research Methods
Week 1 Causal inference
Lecture 1
Causal inference study
The effect of X on Y so the effect of something on something else.
Problems
Small sample (only 41 women where included in this study).
o Is this always a problem? No, it also has a relationship with the effect or with the
outcomes and the homogeneity in the outcomes.
Study performed or financed by commercial company.
o Is this always fatal? This is always a challenge, but you can do a few things to protect
your independence as a researcher.
No control group.
o This is indeed an essential omission.
o What without treatment?
o Potential regression towards the mean (would it also be better without the
intervention? We don’t know).
What do we want to know?
Not interested in the outcome per se: ‘how many imperfections’.
Interested in the role of treatment in achieving this outcome: ‘less imperfections than without
True Match Minerals’.
Conclusion: no meaningful causal conclusion can be drawn from this study.
Causation: towards a usable, formal framework
Formal definition (Hernàn/Robins): “in an individual, a treatment has a causal effect if the
outcome under treatment 1 would be different from the outcome under treatment 2”. Then you
can say:
o Wat would have happened?
o What will happen?
So in the case about True Match Minerals:
Women A uses True Match Minerals: 2 bad spots.
Women A does not use True Match Minerals: 5 bad spots.
Individual treatment effect: 3 spots (60%).
Average treatment effect: average of all individual effects in a population.
Potential outcomes | 1
Causal effect:
Y = outcome.
A = treatment.
1 = yes (received treatment).
0 = no (received no treatment).
I = individual.
Does not equal.
This says that the potential outcomes are different.
Potential outcomes | 2
1
, Advanced Research Methods
Let’s improve the experiment and add a control group.
The outcome of user K with Ya = 1 = 1 (improvement with
treatment).
The outcome of user K with Ya = 0 = 0 (no improvement
without treatment).
Treatment effect for K: 1 – 0 = 1.
Average treatment effect is the average of the last column.
Not all potential outcomes are observed
Counterfactual outcome: potential outcome that is not observed because the subject did not
experience the treatment (‘counter the fact’).
o It could have happened, but it didn’t. For user K we
only see the outcome with treatment. For user M and
N we only see the outcome without problem. This is
the fundamental problem of causal effects.
Potential outcome (Ya=1) is fractural for some objects, and
counterfactual for others.
Fundamental problem
Individual causal effect cannot be observed. Except under extremely strong (and generally
unreasonable) assumptions.
o You don’t know what would happened without the treatment.
Average causal effect cannot be inferred from individual estimates. Causal inference as a
missing data problem.
We need a different approach to causal effects.
Identifiability conditions
Average causal effects can still be determined under certain conditions.
‘Observing’ the counterfactual: what would have happened?
Based on population averages, causal effects can be estimated if three identifiability
conditions hold:
o Positivity.
o Consistency.
o Exchangeability.
If the conditions are met, then association of exposure and outcome is unbiased estimate of
causal effect.
Simple example
Go to the Lijnbaan in Rotterdam.
We ask everyone ‘are you carrying a cigarette lighter?’
Come back after twenty years: who is healthier?
Causal questions: what is the effect of carrying a lighter on health?
Positivity | 1
Observe ‘what would have happened if…’.
This is about the sample and the way it was composed.
‘Positive probability’ of being assigned to each of the treatment levels.
Units are assigned to all relevant ‘treatments’:
o People with and people without a cigarette lighter.
o People with lighter could also not have had a lighter, and vice verca.
o This was not the case in the example of L’oreal: 100% was assigned to True Match
Mineral, 0% to comparison group. Users could not not have used it.
Control group.
2
, Advanced Research Methods
Consistency
Observe ‘what would have happened if…’.
Define ‘if’: clear definition of ‘treatments’.
Hernàn: does water kill? What do you mean by water? You have to be very specific.
Is it consistent?
o How healthy is broccoli? But how much? How often? What else are you eating? What
are we comparing this to?
o Effect of obesity on health? Obesity is not informative. We have to know how people
did or did not have obesity.
o Effect of obesity on job prospects? In this question it doesn’t matter how people got
obeses. This is a consistent definition.
o Effect of healthcare spending on mortality? What is the money going to be spend on?
How would it be spend without? This isn’t a consistent definition.
o Carrying a cigarette lighter? This might be (in hand or in pocket).
Exchangeability | 1
Observe ‘what would have happened if…’.
Treatment groups are exchangeable: it does not matter who gets treatment A and who gets
treatment B.
Notation:
‘Potential outcomes are independent of the treatment that was actually received’.
Are people with and without lighters exchangeable (similar in other respects).
Exchangeability | 2
Observe ‘what would have happened if…’.
It may be necessary to take other factors into account (adjustment).
o Within smoking group, are people with and without lighter exchangeable?
o Within non-smoking group, are people with and without lighter exchangeable?
Association can be ascribed to treatment effect.
Positivity | 2
Units are assigned to all relevant ‘treatments’ within levels of adjustment factors.
‘Positive probability’ of being assigned to each of the treatment levels. We need:
o Smokers with cigarette lighter.
o Smokers without cigarette lighter.
o Non-smokers with cigarette lighter.
o Non-smokers without cigarette lighter.
Stratification
This is when you divide the sample in different groups (according to the value of one variable). In this
case the variable is smoking.
First without stratification: with lighter
65 people are healthy, without lighter 165
people are healthy. We don’t believe we
achieved exchangeability.
With stratification: 50% of smokers is
healthy with lighter and 50% without
lighter. There is within the smoking
group no association with health. Neither
with the non-smoking group.
We had consistency (we know what we mean by
carrying a ligher), we had positivity (enough observations in the different groups) and we achieved
exchangeability (if smoking is the only things that counts). If we believe all the condition are met (we
3
, Advanced Research Methods
cannot measure) then we have an unbiased estimate of the causal effect of carrying a lighter. In this
case the effect is 0 (there is no effect).
Meeting the conditions: Randomized Controlled Trial (RCT)
Select patients.
Randomly assign to the treatment groups.
o Random: exchangeability.
o Random: positivity.
o Consistency: you have to identify the interventions so you have to know what these
groups are.
RCTs versus observational studies
RCT
Limited generalizability (external validity) due to treatment protocol and patient selection.
o You treat people in very specific circumstances (controlled), what does this say about
the real-world?
Practical, ethical considerations.
Observational (non-randomised) study
Real-world outcomes (advantage).
Availability of data (can be collected or are available).
Internal validity threatened by lack of exchangeability.
Positivity and consistency need explicit attention.
Association does not equal causation
In many cases, we are interested in causal effects, not just associations.
Association: statistical relationship.
Causation: difference between potential outcomes.
This association equals this difference if identifiability conditions hold.
We need:
o Theory/subject knowledge.
o Causal structure.
And we design the analysis accordingly.
Adjustment to improve exchangeability
Small number of factors? Stratification is possible. Other ways:
o Matching.
o Weighting.
o Regression analysis.
Complete and correct adjustment leads to exchangeability.
So … what should you adjust for in your analysis?
Traditional selection strategies
Correlation matrix: select variables with sign. association with outcome.
Stepwise backward selection:
o Start with all variables in regression model.
o Remove the variable that is the least statistically significant.
o Repeat steps.
Or: retain variable if removal leads to substantial change in effect estimate (PC lab).
Adjust for confounders, which are defined as being:
o Associated with the exposure, and;
o Conditionally associated with the outcome, given the exposure;
o Not in the causal pathway between exposure and outcome.
These are alternatives, but don’t.
4
