Quantitative research methods
Lecture 1 – Designing research
Quantitative research
Quantitative research > quantitative research methods revolve around answering a particular research
question by collecting numerical data that are analysed using mathematical methods > in particular, statistics
- Experimental data > research that uses structured observations from the real world to attempt to
answer question > by observations
o Often collected in laboratory environments in the natural sciences
o Much more difficult in the social sciences > here we mainly focus on analysing non-
experimental (observational) data
Meaning > not accumulated through controlled experiments
Measurements are hard to take precisely
- Types
o Descriptive research > what?
Interested in a quantitative answer > which program has the most students?
Interested in a numerical change > are the numbers of premaster students in our
university rising compared to last year?
o Inferential research > why?
Test relationships > what is the relation between self-esteem and average grade?
Explain something > what factors cause changes in student performance over time?
- Be careful because numbers and statistics are seldom objective and disinterested
o Data have always been manipulated and sometimes with the wrong intentions
- Why do we need quantitative research?
o In essence, quantitative research methods provide us with a toolbox to study the (social)
world around us using the scientific method
o Helps in minimizing cognitive assumptions that may distort our interpretation
o Depending on the state of prior theory and research on the topic, you must use quantitative
methods to make a useful contribution to our understanding of the world
o Only way to establish causal relationships
- Simpson’s paradox refers to a phenomenon whereby
the association between a pair of variables (X, Y)
reverses sign upon conditioning of a third variable, Z,
regardless of the value taken by Z
o Example > a higher dosage of medicine may be
associated with higher recovery rates at the
population-level > however, within subgroups
(e.g., for both males and females) a higher
dosage my results in lower recovery rates
,Research design
Research Design > tells you how you will answer a question > good research design can answer this question
Theory > identifies what variables are important and for what reasons > how they are interrelated and why
and identifies conditions under which they should be related and not related
- Variable > when we end up with when we apply our measure to something in the world
o That is, variables are the actual data that we end up with in our data sets
- In any relationship > you will have independent variable/predictor and a dependent variable/outcome
o Education and wage > there will be a positive or negative correlation > here, a control
variable could be age or gender
- Hypothesis > defines the expected relationship between variables and therewith tests the existence
of a theory > variable X has a positive relationship with variable Y
Research questions
Research question in quantitative empirical research > a question that can be answered
- What the best movie is > cannot be answered because “best” is ambiguous > which movie is sold the
most > correct), and for which having that answer will improve understanding of how the world works
- Good research question
o Takes us from theory to hypothesis
o A hypothesis is a specific statement about what we will observe in the world
Data mining > just look at patterns in data without using a question
- However, that is helpful to find patterns and make predictions under stability (stability > the process
giving us the data does not change)
- Why does data mining have difficulty helping theory?
o Focuses on what is in the data > not why > fantastic at revealing correlations, but the
correlations it uncovers may have little to do with causality or an understanding of why those
variables move together
- Trying to understand ice cream sales, you notice that the proportion of people wearing shorts is a
good predictor of ice cream sales > but shorts-wearing is not why people buy ice cream > because it is
hot > for a data miner, the shorts and ice cream connection is compelling
- Less good at improving theory > why?
o Data mining is good at finding relationships but not at telling why relationships are there >
focuses on what is in the data and not why
o Data mining also does not deal in abstraction
o High chances of false positives
Without a research question there is no reason not to just check everything >
something is going to pop up as related by random chance if you check enough stuff
- Data mining is not all bad > plenty of theories come from looking at the data in the first place, noticing
a pattern, and wonder why the pattern is showing up
o The responsible thing to do at that point is to not just take the pattern as given > instead,
take the pattern and look to see if it pops up in other data (if it is replicated)
Research question > tells us a hypothesis to test, such that the result of that test tells us something about the
theory > how do you know if your research question is a good one?
- Consider potential results
o Imagine what kind of sense you would make of that result > can you link results t theory?
- Consider feasibility
o Should be a question that can be answered using right data > check if right data is available
- Consider scale
o What kind of resource and time can you dedicate to answering the research question?
- Consider design
o Figure out if there is a reasonable research design you can use to answer it
- Keep it simple
, o Do not bundle a bunch of research questions into one
Lecture 2 – Research design
Theory building
What is a legitimate, value-added contribution to theory development? > new theory is not generated from
scratch > work on improving what already exists > three important points
- Proposed improvements should focus on multiple elements of a theory > adds the quality of
completeness and thoroughness to theoretical work
- Theoretical critiques should marshal compelling evidence > this evidence can be logical, empirical, or
epistemological
- Theoretical critiques should propose remedies or alternatives
Theory > an explanation of relationships among concepts or events within a set of boundary conditions
- A good theory simplifies and explains complex real—world phenomena
o Relationship between 2+ variables > hypothesis
- Elements of a good theory
o What
Which factors (variables, concepts) logically should be considered as part of the
explanation of the social or individual phenomena of interest?
Criteria for the right factors
Comprehensiveness > are all relevant factors included?
Parsimony > should some factors be deleted because they add little
additional value to our understanding?
o How
How are the factors related? > the links between constructs and variables
Involves using arrows to connect the boxes
Introduces causality
A visual representation often clarifies the author’s thinking
o Why
The glue that justifies the selection of constructs/variables and their proposed
relationships
Why explains, what and how describe
o Who, where, when?
The conditions under which the theory should hold
Set the boundaries of generalizability
Moderators
Typically, the relationship between one variable and another is different for
different people, in different contexts, or at different points in time
o E.g., The impact of education on your wage is stronger for young
people than for older people
We say that the relationship between x and y depends on a third variable z,
the moderator
o E.g., age is the moderator in the example above
, Moderators help us identify the boundary conditions of our theory
What factors are considered in judging conceptual papers?
- What is new?
o Significant, value-added contribution
- So what?
o Will the theory likely change the practice of organizational science in this area?
o Does the paper go beyond making statements?
- Why so?
o Are the underlying logic and supporting evidence compelling?
o Theory development should be built on a foundation of convincing argumentation and
grounded in reasonable, explicit views of human nature and organizational practice
o The “glue” that justifies the selection of variables and their proposed relationships
- Well done?
o Reflect second-thinking, conveying completeness and thoroughness
o Are multiple theoretical elements covered?
- Done well?
o Well written, logic flow
- Why now?
o Is the topic of contemporary interest to scholars in this area?
- Who cares?
o Make a significant contribution to current thinking
Example > phenomenon of interest > employee skills
- Research question > the impact of top management cognitive abilities on the skill level of employees
o Proposition > top management cognitive abilities increase the skill level of employees
o Hypothesis > The level of top management education increases the share of skilled in
employees in the company
o What
Constructs: TM cognitive ability, employee skills
Variables: TM education level (explanatory or independent variable), share of skilled
workers (outcome or dependent variable)
o How
Proposition: Top management cognitive abilities increase the skill level of
employees
Hypothesis: The level of top management education increases the share of skilled in
employees in the company
o Why
Assortative matching (“birds of a feather flock
together”)
o Who, where, when?
When the manager is inexperience
The Data Generating Process
The DGP > we assume there are similar “laws” that govern the social world around us
- We call these “laws” the data generating process (DGP)
- Your theoretical model is only a (small) part of the broader, more complex data generating process
- We need to make credible claims about the complete DGP so that we can identify the variation in the
data that answers our research question
- Scientists believe that there are regular laws that govern the way the universe works > these laws are
an example of a data generating process > the laws work behind the scenes > we cannot see the data,
but we do see the data that result from them
o The data generating process (DGP) is divided into two parts
The parts we know and
The parts we don’t know > what we are hoping to learn about with our research
Lecture 1 – Designing research
Quantitative research
Quantitative research > quantitative research methods revolve around answering a particular research
question by collecting numerical data that are analysed using mathematical methods > in particular, statistics
- Experimental data > research that uses structured observations from the real world to attempt to
answer question > by observations
o Often collected in laboratory environments in the natural sciences
o Much more difficult in the social sciences > here we mainly focus on analysing non-
experimental (observational) data
Meaning > not accumulated through controlled experiments
Measurements are hard to take precisely
- Types
o Descriptive research > what?
Interested in a quantitative answer > which program has the most students?
Interested in a numerical change > are the numbers of premaster students in our
university rising compared to last year?
o Inferential research > why?
Test relationships > what is the relation between self-esteem and average grade?
Explain something > what factors cause changes in student performance over time?
- Be careful because numbers and statistics are seldom objective and disinterested
o Data have always been manipulated and sometimes with the wrong intentions
- Why do we need quantitative research?
o In essence, quantitative research methods provide us with a toolbox to study the (social)
world around us using the scientific method
o Helps in minimizing cognitive assumptions that may distort our interpretation
o Depending on the state of prior theory and research on the topic, you must use quantitative
methods to make a useful contribution to our understanding of the world
o Only way to establish causal relationships
- Simpson’s paradox refers to a phenomenon whereby
the association between a pair of variables (X, Y)
reverses sign upon conditioning of a third variable, Z,
regardless of the value taken by Z
o Example > a higher dosage of medicine may be
associated with higher recovery rates at the
population-level > however, within subgroups
(e.g., for both males and females) a higher
dosage my results in lower recovery rates
,Research design
Research Design > tells you how you will answer a question > good research design can answer this question
Theory > identifies what variables are important and for what reasons > how they are interrelated and why
and identifies conditions under which they should be related and not related
- Variable > when we end up with when we apply our measure to something in the world
o That is, variables are the actual data that we end up with in our data sets
- In any relationship > you will have independent variable/predictor and a dependent variable/outcome
o Education and wage > there will be a positive or negative correlation > here, a control
variable could be age or gender
- Hypothesis > defines the expected relationship between variables and therewith tests the existence
of a theory > variable X has a positive relationship with variable Y
Research questions
Research question in quantitative empirical research > a question that can be answered
- What the best movie is > cannot be answered because “best” is ambiguous > which movie is sold the
most > correct), and for which having that answer will improve understanding of how the world works
- Good research question
o Takes us from theory to hypothesis
o A hypothesis is a specific statement about what we will observe in the world
Data mining > just look at patterns in data without using a question
- However, that is helpful to find patterns and make predictions under stability (stability > the process
giving us the data does not change)
- Why does data mining have difficulty helping theory?
o Focuses on what is in the data > not why > fantastic at revealing correlations, but the
correlations it uncovers may have little to do with causality or an understanding of why those
variables move together
- Trying to understand ice cream sales, you notice that the proportion of people wearing shorts is a
good predictor of ice cream sales > but shorts-wearing is not why people buy ice cream > because it is
hot > for a data miner, the shorts and ice cream connection is compelling
- Less good at improving theory > why?
o Data mining is good at finding relationships but not at telling why relationships are there >
focuses on what is in the data and not why
o Data mining also does not deal in abstraction
o High chances of false positives
Without a research question there is no reason not to just check everything >
something is going to pop up as related by random chance if you check enough stuff
- Data mining is not all bad > plenty of theories come from looking at the data in the first place, noticing
a pattern, and wonder why the pattern is showing up
o The responsible thing to do at that point is to not just take the pattern as given > instead,
take the pattern and look to see if it pops up in other data (if it is replicated)
Research question > tells us a hypothesis to test, such that the result of that test tells us something about the
theory > how do you know if your research question is a good one?
- Consider potential results
o Imagine what kind of sense you would make of that result > can you link results t theory?
- Consider feasibility
o Should be a question that can be answered using right data > check if right data is available
- Consider scale
o What kind of resource and time can you dedicate to answering the research question?
- Consider design
o Figure out if there is a reasonable research design you can use to answer it
- Keep it simple
, o Do not bundle a bunch of research questions into one
Lecture 2 – Research design
Theory building
What is a legitimate, value-added contribution to theory development? > new theory is not generated from
scratch > work on improving what already exists > three important points
- Proposed improvements should focus on multiple elements of a theory > adds the quality of
completeness and thoroughness to theoretical work
- Theoretical critiques should marshal compelling evidence > this evidence can be logical, empirical, or
epistemological
- Theoretical critiques should propose remedies or alternatives
Theory > an explanation of relationships among concepts or events within a set of boundary conditions
- A good theory simplifies and explains complex real—world phenomena
o Relationship between 2+ variables > hypothesis
- Elements of a good theory
o What
Which factors (variables, concepts) logically should be considered as part of the
explanation of the social or individual phenomena of interest?
Criteria for the right factors
Comprehensiveness > are all relevant factors included?
Parsimony > should some factors be deleted because they add little
additional value to our understanding?
o How
How are the factors related? > the links between constructs and variables
Involves using arrows to connect the boxes
Introduces causality
A visual representation often clarifies the author’s thinking
o Why
The glue that justifies the selection of constructs/variables and their proposed
relationships
Why explains, what and how describe
o Who, where, when?
The conditions under which the theory should hold
Set the boundaries of generalizability
Moderators
Typically, the relationship between one variable and another is different for
different people, in different contexts, or at different points in time
o E.g., The impact of education on your wage is stronger for young
people than for older people
We say that the relationship between x and y depends on a third variable z,
the moderator
o E.g., age is the moderator in the example above
, Moderators help us identify the boundary conditions of our theory
What factors are considered in judging conceptual papers?
- What is new?
o Significant, value-added contribution
- So what?
o Will the theory likely change the practice of organizational science in this area?
o Does the paper go beyond making statements?
- Why so?
o Are the underlying logic and supporting evidence compelling?
o Theory development should be built on a foundation of convincing argumentation and
grounded in reasonable, explicit views of human nature and organizational practice
o The “glue” that justifies the selection of variables and their proposed relationships
- Well done?
o Reflect second-thinking, conveying completeness and thoroughness
o Are multiple theoretical elements covered?
- Done well?
o Well written, logic flow
- Why now?
o Is the topic of contemporary interest to scholars in this area?
- Who cares?
o Make a significant contribution to current thinking
Example > phenomenon of interest > employee skills
- Research question > the impact of top management cognitive abilities on the skill level of employees
o Proposition > top management cognitive abilities increase the skill level of employees
o Hypothesis > The level of top management education increases the share of skilled in
employees in the company
o What
Constructs: TM cognitive ability, employee skills
Variables: TM education level (explanatory or independent variable), share of skilled
workers (outcome or dependent variable)
o How
Proposition: Top management cognitive abilities increase the skill level of
employees
Hypothesis: The level of top management education increases the share of skilled in
employees in the company
o Why
Assortative matching (“birds of a feather flock
together”)
o Who, where, when?
When the manager is inexperience
The Data Generating Process
The DGP > we assume there are similar “laws” that govern the social world around us
- We call these “laws” the data generating process (DGP)
- Your theoretical model is only a (small) part of the broader, more complex data generating process
- We need to make credible claims about the complete DGP so that we can identify the variation in the
data that answers our research question
- Scientists believe that there are regular laws that govern the way the universe works > these laws are
an example of a data generating process > the laws work behind the scenes > we cannot see the data,
but we do see the data that result from them
o The data generating process (DGP) is divided into two parts
The parts we know and
The parts we don’t know > what we are hoping to learn about with our research
