Hoorcolleges Advanced Research Methods
College 1 (dinsdag 8-11-2021)
What is ARM about?
- Give you tools to construct a methodologically sound research design.
- Be able to critically judge the articles of others. We especially look at causality. How can you
convincedly say x leads to y?
ARM gives a broad overview, the goal is to have a basic understanding, so you have to know:
- The principles
- The usefulness, when to use it
- Strengths and limitation (causality)
ARM does not aim at:
- Full mastery of the methods
- Providing you with specific practical skills in applying the methods
Exam: written exam and 100% of the final grade on campus.
This course is about causal inference:
Inference = the goal of social science is “to infer beyond the immediate data to something broader
that is not directly observed”.
Social scientists want to say something about reality, but we are not observing the whole reality.
Inference is the step we make from our data to reality. We are generalizing what we find in our data
to the world. We infer that something is true based on our data.
We distinguish two types of inference:
1. Quantitative: from a random sample to generalize/infer to the whole population
2. Qualitative: from one or several cases in dept to broader cases
Another distinction:
1. Descriptive: we need to know what the phenomenon actually is that we want to explain. We
first need to know what Y actually looks like. We have to show what is happening, before
showing why it is happening (Goldthorpe, 2001).
2. Causal: in social sciences we usually look for explanation (XY)
Descriptive questions can look like:
- How high is election turnout in different states of the US?
- How did election turnout in the US develop over the past three decades?
Causal questions can look like:
- How do economic conditions impact cross-state differences in turnout?
- What is the effect of social media use on the decision to vote?
But causal question can also be about the mechanism between x and y:
- Through which mechanism do social media use influence people’s voting decision?
There are different conceptualizations of causality. Goldthorpe give three in his text.
First, causation as robust dependence:
“while correlation (…) does not imply causation, causation must in some way or other imply
association”. This means if X is really causally related to Y, at minimum we should see that they are
correlated. The two variables are dependent on each other.
,Correlation=covariance=association
Robust: correlation remains after controlling for possible other explanations (cofounders).
If the effect of income on trust disappears when you include age it is a spurious associated. The bias
that exists in the income trust relation, when you don’t control for age, you have a omitted variable
bias.
Typically the robust dependent approach uses regression models. You add the relation you are
interested in and then add all the possible control variables that explain this relationship (so which
are related to both variables).
Partioling out approach = robust dependence approach
Critiques on the robust dependence approach:
1. Cofounders are always possible, it will never be fully satisfied.
- Alternative: consequential manipulation. This is based on counterfactual reasoning: observe
Y in the presence and absence of X. but you can never observe the exact same situation twice
so this method is critiqued as well
- Effects of causes: causes must be manipulable. This way is the other way around. Problem of
agency
2. What is the mechanism?
- You might have robust casualization, but why does this correlation take place?
Alternative: causation as generative process (mechanism-based approach)
Causal mechanisms
Causal mechanism is a theoretical account that shows how specific outcomes/events or empirical
regularities come about. It shows the structure of the causal process. It shows the logical connection
between X and Y. it is an answer to a why question.
A causal mechanism is the pathway(s) through which X might affect Y.
It is a story about the arrow between x and y.
Can or should we empirically study causal mechanisms or is it just a story?
At minimum: theoretical specification (story)
If it is possible you have to empirically test the mechanism. You can test in between steps. A
causal chain.
Mechanism require some form of methodological individualism.
Everything that happens in the social world is a consequence of individual actions.
Methodological individualism(s):
Social phenomena should be explained as resulting from the behavior of individual actors. These
behaviors should be explained as motivated by intentional states of the actors.
Why are there multiple versions of methodological individualism(s)?
There are different versions. They refer to how strongly they rely on individual action. The weak
version can be called structural individualism and is the most used version (outlook in this course as
well).
The strong version will treat social phenomena as irrelevant or strictly endogenous. Social
phenomena are an explanandum, but not an explanans.
,The weak version see the social phenomena as fundamental importance to understand why actors
act the way they act. They are both endogenous and exogenous. Social phenomena are understood
as the intended and unintended consequences of actions of individuals actors. But action is shaped a
pre-existing social context (structural individualism).
Coleman’s bathtub (again)…………….
You could see the bathtub as a certain causal chain.
Example:
Methodological individualism gives us micro-foundations for macro-relations.
This way of explanation requires an account of how actors act (theory of action). But does not
stipulate any specific theory of action. Could be:
- A rational choice theory
- Social identity theory
- Behavioral theory
Kittel gives critiques on the macro-quantitative research.
Can we establish causal relationships between macro social phenomena by studying them only on
the macro level?
Quantitative comparative research is a method by which hypotheses about correlations between
variables are tested on data aggregated and measured at the level of nation-states.
It is important to know what the macro-level is, before reading this paper.
1. D
2. A
3. B
College 2 (donderdag 11-11-2021)
, Micro and macro data might look similar. You have to ask yourself what the unit of analysis is.
Descriptive properties of individuals vs. descriptive properties of groups.
What is Robinsons argument?
Based on macro level data, you draw inferences on the micro level. Inferences might be false this
way.
Ecological/between group correlation
Individual/within group correlations
Inverse: atomistic fallacy: you have micro data and draw inferences on the macro level. Inferences
might be false this way.
Kittel:
Crucial point: there is a crucial distinction between macro level that is aggregated individual behavior
(model behavior pattern) and collective behaviors (strategic interaction).
Causal heterogeneity: there might be a difference between different individuals, so between men
and women.
See assignment 1!
Concluding remarks:
You should have data at the level of your theory, ideally. Of course this is often very difficult,
especially when you work at the micro level/individuals and when you do cross-country or cross-time
analysis. This is why researchers do research on the macro-level a lot. But you have to be aware of
this. you can compensate this by also doing a smaller micro level research.
It is expensive to collect micro-data. You just have to be critical while using this.
College 3 (dinsdag 16-11-2021) multilevel modeling
Last week: research looks at the macro level a lot, when they actually want to say something about
the micro level.
Now we have a method that can incorporate the macro and micro level in one analysis.
What are multi-level data structures?
It is about data that is nested: observations that are nested in groups. Much of the data that we have
is clustered in a way.
Nested = hierarchical = clustered different terms for the same thing. The word we work with is
people nested I countries.
People that are in the same country are believed to share the same characteristics.
It is also used a lot in educational studies: children nested in classes. Etc. many examples to give.
Persons are denoted as I and countries as J normally. In the example dataset are multiple layers. Age
varies within the countries, same goes for satisfaction with government. These are individual/micro
level variables. GDP growth is the same for every person because it is a country level characteristic. It
only varies between the countries. The same goes for number of countries in parliament.
So,
Variables varying within groups = micro level
Variables that only vary between groups = macro
Then we have multi-level data, but what should we do with it? They pose a specific statistical
problem, because the observations are clustered.
College 1 (dinsdag 8-11-2021)
What is ARM about?
- Give you tools to construct a methodologically sound research design.
- Be able to critically judge the articles of others. We especially look at causality. How can you
convincedly say x leads to y?
ARM gives a broad overview, the goal is to have a basic understanding, so you have to know:
- The principles
- The usefulness, when to use it
- Strengths and limitation (causality)
ARM does not aim at:
- Full mastery of the methods
- Providing you with specific practical skills in applying the methods
Exam: written exam and 100% of the final grade on campus.
This course is about causal inference:
Inference = the goal of social science is “to infer beyond the immediate data to something broader
that is not directly observed”.
Social scientists want to say something about reality, but we are not observing the whole reality.
Inference is the step we make from our data to reality. We are generalizing what we find in our data
to the world. We infer that something is true based on our data.
We distinguish two types of inference:
1. Quantitative: from a random sample to generalize/infer to the whole population
2. Qualitative: from one or several cases in dept to broader cases
Another distinction:
1. Descriptive: we need to know what the phenomenon actually is that we want to explain. We
first need to know what Y actually looks like. We have to show what is happening, before
showing why it is happening (Goldthorpe, 2001).
2. Causal: in social sciences we usually look for explanation (XY)
Descriptive questions can look like:
- How high is election turnout in different states of the US?
- How did election turnout in the US develop over the past three decades?
Causal questions can look like:
- How do economic conditions impact cross-state differences in turnout?
- What is the effect of social media use on the decision to vote?
But causal question can also be about the mechanism between x and y:
- Through which mechanism do social media use influence people’s voting decision?
There are different conceptualizations of causality. Goldthorpe give three in his text.
First, causation as robust dependence:
“while correlation (…) does not imply causation, causation must in some way or other imply
association”. This means if X is really causally related to Y, at minimum we should see that they are
correlated. The two variables are dependent on each other.
,Correlation=covariance=association
Robust: correlation remains after controlling for possible other explanations (cofounders).
If the effect of income on trust disappears when you include age it is a spurious associated. The bias
that exists in the income trust relation, when you don’t control for age, you have a omitted variable
bias.
Typically the robust dependent approach uses regression models. You add the relation you are
interested in and then add all the possible control variables that explain this relationship (so which
are related to both variables).
Partioling out approach = robust dependence approach
Critiques on the robust dependence approach:
1. Cofounders are always possible, it will never be fully satisfied.
- Alternative: consequential manipulation. This is based on counterfactual reasoning: observe
Y in the presence and absence of X. but you can never observe the exact same situation twice
so this method is critiqued as well
- Effects of causes: causes must be manipulable. This way is the other way around. Problem of
agency
2. What is the mechanism?
- You might have robust casualization, but why does this correlation take place?
Alternative: causation as generative process (mechanism-based approach)
Causal mechanisms
Causal mechanism is a theoretical account that shows how specific outcomes/events or empirical
regularities come about. It shows the structure of the causal process. It shows the logical connection
between X and Y. it is an answer to a why question.
A causal mechanism is the pathway(s) through which X might affect Y.
It is a story about the arrow between x and y.
Can or should we empirically study causal mechanisms or is it just a story?
At minimum: theoretical specification (story)
If it is possible you have to empirically test the mechanism. You can test in between steps. A
causal chain.
Mechanism require some form of methodological individualism.
Everything that happens in the social world is a consequence of individual actions.
Methodological individualism(s):
Social phenomena should be explained as resulting from the behavior of individual actors. These
behaviors should be explained as motivated by intentional states of the actors.
Why are there multiple versions of methodological individualism(s)?
There are different versions. They refer to how strongly they rely on individual action. The weak
version can be called structural individualism and is the most used version (outlook in this course as
well).
The strong version will treat social phenomena as irrelevant or strictly endogenous. Social
phenomena are an explanandum, but not an explanans.
,The weak version see the social phenomena as fundamental importance to understand why actors
act the way they act. They are both endogenous and exogenous. Social phenomena are understood
as the intended and unintended consequences of actions of individuals actors. But action is shaped a
pre-existing social context (structural individualism).
Coleman’s bathtub (again)…………….
You could see the bathtub as a certain causal chain.
Example:
Methodological individualism gives us micro-foundations for macro-relations.
This way of explanation requires an account of how actors act (theory of action). But does not
stipulate any specific theory of action. Could be:
- A rational choice theory
- Social identity theory
- Behavioral theory
Kittel gives critiques on the macro-quantitative research.
Can we establish causal relationships between macro social phenomena by studying them only on
the macro level?
Quantitative comparative research is a method by which hypotheses about correlations between
variables are tested on data aggregated and measured at the level of nation-states.
It is important to know what the macro-level is, before reading this paper.
1. D
2. A
3. B
College 2 (donderdag 11-11-2021)
, Micro and macro data might look similar. You have to ask yourself what the unit of analysis is.
Descriptive properties of individuals vs. descriptive properties of groups.
What is Robinsons argument?
Based on macro level data, you draw inferences on the micro level. Inferences might be false this
way.
Ecological/between group correlation
Individual/within group correlations
Inverse: atomistic fallacy: you have micro data and draw inferences on the macro level. Inferences
might be false this way.
Kittel:
Crucial point: there is a crucial distinction between macro level that is aggregated individual behavior
(model behavior pattern) and collective behaviors (strategic interaction).
Causal heterogeneity: there might be a difference between different individuals, so between men
and women.
See assignment 1!
Concluding remarks:
You should have data at the level of your theory, ideally. Of course this is often very difficult,
especially when you work at the micro level/individuals and when you do cross-country or cross-time
analysis. This is why researchers do research on the macro-level a lot. But you have to be aware of
this. you can compensate this by also doing a smaller micro level research.
It is expensive to collect micro-data. You just have to be critical while using this.
College 3 (dinsdag 16-11-2021) multilevel modeling
Last week: research looks at the macro level a lot, when they actually want to say something about
the micro level.
Now we have a method that can incorporate the macro and micro level in one analysis.
What are multi-level data structures?
It is about data that is nested: observations that are nested in groups. Much of the data that we have
is clustered in a way.
Nested = hierarchical = clustered different terms for the same thing. The word we work with is
people nested I countries.
People that are in the same country are believed to share the same characteristics.
It is also used a lot in educational studies: children nested in classes. Etc. many examples to give.
Persons are denoted as I and countries as J normally. In the example dataset are multiple layers. Age
varies within the countries, same goes for satisfaction with government. These are individual/micro
level variables. GDP growth is the same for every person because it is a country level characteristic. It
only varies between the countries. The same goes for number of countries in parliament.
So,
Variables varying within groups = micro level
Variables that only vary between groups = macro
Then we have multi-level data, but what should we do with it? They pose a specific statistical
problem, because the observations are clustered.
