The Effect – Nick Huntington-Klein
Chapter 1 – designing research
Quantum mechanics = perhaps the most precise scientific field in existence, with
accurate measurements and predictions out to more than a dozen decimal places. But
even then, ‘accurate to fourteen decimal places’ isn’t the same thing as ‘accurate’
Empirical research = any research that uses structured observations from the real
world to attempt to answer questions
Quantitative empirical research = empirical research that uses quantitative
measurements (numbers, usually). More data sets, fewer interviews --> can be tricky,
measurements are hard to take precisely or interpret accurately
Problem: the numbers that we want to observe often don’t tell us exactly what we want
to know --> needs right kind of analysis that will answer the question
Chapter 2 – research questions
Research question = a question that can be answered, and for which having that
answer will improve your understanding of how the world works
- That can be answered: its possible for there to be some set of evidence that, if
you found that evidence, your question would have a believable answer.
- Improve your understanding of how the world works: the research question,
once answered, should tell you about something broader than itself. It should
inform theory in some way. Theory just means that there’s a why or a because
somewhere.
--> a good research question takes us from theory to hypothesis
- Hypothesis: a specific statement about what we will observe in the world
- Research question: should be something that, if you answer it, helps improve
your why explanation. Great research questions often come from the theory
themselves
Does a research question inform theory
- Find an unexpected result, and then wonder whether it would make you change
your understanding of the world
- A really good research question, once answered, should be hard to explain away
just because its inconvenient
Data mining = skip the part of deriving a research question from a theory and instead
just see what sort of patterns are in the data --> go to the data, look for patterns, and
report back (a lot of the field of data science, but data mining can be done any time you
have some data. Look at the data, see what’s in there, and work backwards)
,-- > good for: finding patterns, making predictions under stability (stability: the process
giving us the data doesn’t change) --> very valuable
- Doing something that just asks what we see rather than why is the right angle to
take there + patterns in data can give us ideas for research questions that we
can examine further in other data sources + best angle to take when we don’t
care about why
--> less good for: improving our understanding (helping improve theory) and has
tendency to find false positives
Why does data mining have difficulty helping theory?
- Data mining focuses on what’s in the data, not why its in the data --> fantastic at
revealing correlations (patterns in data of how variables we’ve observed have
varied together in the past) but the correlations have little to do with causality, or
an understanding of why those variables move together -->
o Data mining is well-equipped to find the relationship but poorly equipped
to tell us why that relationship is there
- Data mining is so focused on data that it doesn’t really deal in abstraction.
There’s just a flat bit and some straight-up-and-down bits underneath the flat bit
(chair example). Data mining would miss why (because it allows us to sit on it).
- False positives: something pops up as looking related, just by random chance --
> unlikely to pop up again --> tools in data science for avoiding false positives:
cross-validation, training and testing sets
Data mining isn’t all bad
- Plenty of theories come from looking at the data in the first place, noticing a
pattern, and wondering why the pattern is showing up, or whether the pattern is
even real
- The responsible thing to do is to not just take the patterns as given --> that’s
where the data mining problem is
- Is just bad as a final step if you’re trying to explain the world --> can still work as
a source of ideas
Where do research questions come from = curiosity, we want to know how the world
works
--> 2 steps: thinking about theory, and coming up with a research question (either one
can come first)
- Begins with theory: the process continues with the hypothesis and might lead to
the research questions. The research questions tell us a hypothesis to test such
that the result of that test tells us something about the theory
- Begins with question: we might wonder after the question why we came up with
such an idea in the first place or we might wonder if we answered the question
leading us back to the theory --> if you can’t figure out why you would ask the
, question, it may not be a great research question or at least you’d have a hard
time getting anyone to care about the answer once you had it
Opportunity = sometimes research questions also come from opportunity --> have a
neat data set? Think about what data is available to you and whether any related
research questions or theories come to mind (try to do this after understanding what is
in the data before actually analyzing the data, unless your goal is data mining)
- Something unusual or interesting happened, you might ask “what research
questions would this allow me to answer?” and from there you have a research
question, and from there a theory!
How do you know if you’ve got a good one? (Check relationship between research
question and theory)
- Consider potential results: consider the potential answers you might get -->
what kind of sense you’d make of that result or what conclusion you can draw --
> what would the results tell you about the variable? --> if you can’t say
something interesting about your potential results, that probably means your
research question and your theory aren’t as closely linked as you think
- Consider feasibility: a research question should be a question that can be
answered using the right data, if the right data is available. But is the right data
available? If answering your research question is possible but requires following
millions of people repeatedly for decades, or trying to measure something that’s
really hard to measure accurately, then that research question might not be
feasible (remember lunch 3 years ago/ getting access to private finances of
thousands unwilling people)
- Consider scale: what kind of resources and time can you dedicate to answering
the research question? --> Given the confines of, say, a term paper, you could
take some wild swings at a question, but you’re likely to do a much more
thorough job answering questions with a lot less complexity.
- Consider design: a research question can be great on its own
- Keep it simple: don’t bundle to much research questions into one
--> consider feasibility, scale, and design and keep it simple
Chapter 3 Description of variables
Description of variables: its important to accurately describe variables in empirical
research.
Variable: bunch of observations of the same measurement (in the context of empirical
research) --> successfully describing a variable = being able to take those observations
and clearly explain what was observed without making someone look through all
scores.
Types of variables: first step is to figure out what kind of variable it is
- Continuous variable: can take any value
, - Count variable: count something, how many times something has happened/
how many of something are there --> can’t be negative and can’t take fractional
values
- Ordinal variables: variables where some values are ‘more’ and others are ‘less’,
but no rule how much more is (final completed level of schooling)
- Categorical variable: recording which category an observation is in = different
options
o Special version of categorical = binary variables: categorical variables
that take 2 values (yes/no)
- Qualitative variables: catch-all category of everything else --> aren’t numeric or
categorical (text of Washington Post headline) --> tend to contain a lot of detail
that resists boiling-down-and-summarizing --> often turned into one of the other
variables above
The distribution
2. **Types of Variables**:
- **Continuous Variables**: These can take any value within a range, like income or
temperature.
- **Count Variables**: These represent counts of occurrences, like the number of
events.
- **Ordinal Variables**: These have a natural order but not a fixed interval between
them, like rankings or scales of satisfaction.
- **Categorical Variables**: These include nominal variables without intrinsic ranking,
like colors or types. Special cases include binary variables, which have only two
categories (e.g., yes/no).
- **Qualitative Variables**: These are variables that aren't numeric or categorical, like
the text of a headline.
3. **Distribution of Variables**: This section describes how to analyze and interpret the
distribution of variables:
- For categorical and ordinal variables, you can use frequency tables or bar charts to
illustrate the distribution.
- For continuous variables, distributions are typically depicted using histograms or
density plots to show the range and concentration of data values.
4. **Summarizing Distributions**: Several statistics are used to summarize
distributions, including:
Chapter 1 – designing research
Quantum mechanics = perhaps the most precise scientific field in existence, with
accurate measurements and predictions out to more than a dozen decimal places. But
even then, ‘accurate to fourteen decimal places’ isn’t the same thing as ‘accurate’
Empirical research = any research that uses structured observations from the real
world to attempt to answer questions
Quantitative empirical research = empirical research that uses quantitative
measurements (numbers, usually). More data sets, fewer interviews --> can be tricky,
measurements are hard to take precisely or interpret accurately
Problem: the numbers that we want to observe often don’t tell us exactly what we want
to know --> needs right kind of analysis that will answer the question
Chapter 2 – research questions
Research question = a question that can be answered, and for which having that
answer will improve your understanding of how the world works
- That can be answered: its possible for there to be some set of evidence that, if
you found that evidence, your question would have a believable answer.
- Improve your understanding of how the world works: the research question,
once answered, should tell you about something broader than itself. It should
inform theory in some way. Theory just means that there’s a why or a because
somewhere.
--> a good research question takes us from theory to hypothesis
- Hypothesis: a specific statement about what we will observe in the world
- Research question: should be something that, if you answer it, helps improve
your why explanation. Great research questions often come from the theory
themselves
Does a research question inform theory
- Find an unexpected result, and then wonder whether it would make you change
your understanding of the world
- A really good research question, once answered, should be hard to explain away
just because its inconvenient
Data mining = skip the part of deriving a research question from a theory and instead
just see what sort of patterns are in the data --> go to the data, look for patterns, and
report back (a lot of the field of data science, but data mining can be done any time you
have some data. Look at the data, see what’s in there, and work backwards)
,-- > good for: finding patterns, making predictions under stability (stability: the process
giving us the data doesn’t change) --> very valuable
- Doing something that just asks what we see rather than why is the right angle to
take there + patterns in data can give us ideas for research questions that we
can examine further in other data sources + best angle to take when we don’t
care about why
--> less good for: improving our understanding (helping improve theory) and has
tendency to find false positives
Why does data mining have difficulty helping theory?
- Data mining focuses on what’s in the data, not why its in the data --> fantastic at
revealing correlations (patterns in data of how variables we’ve observed have
varied together in the past) but the correlations have little to do with causality, or
an understanding of why those variables move together -->
o Data mining is well-equipped to find the relationship but poorly equipped
to tell us why that relationship is there
- Data mining is so focused on data that it doesn’t really deal in abstraction.
There’s just a flat bit and some straight-up-and-down bits underneath the flat bit
(chair example). Data mining would miss why (because it allows us to sit on it).
- False positives: something pops up as looking related, just by random chance --
> unlikely to pop up again --> tools in data science for avoiding false positives:
cross-validation, training and testing sets
Data mining isn’t all bad
- Plenty of theories come from looking at the data in the first place, noticing a
pattern, and wondering why the pattern is showing up, or whether the pattern is
even real
- The responsible thing to do is to not just take the patterns as given --> that’s
where the data mining problem is
- Is just bad as a final step if you’re trying to explain the world --> can still work as
a source of ideas
Where do research questions come from = curiosity, we want to know how the world
works
--> 2 steps: thinking about theory, and coming up with a research question (either one
can come first)
- Begins with theory: the process continues with the hypothesis and might lead to
the research questions. The research questions tell us a hypothesis to test such
that the result of that test tells us something about the theory
- Begins with question: we might wonder after the question why we came up with
such an idea in the first place or we might wonder if we answered the question
leading us back to the theory --> if you can’t figure out why you would ask the
, question, it may not be a great research question or at least you’d have a hard
time getting anyone to care about the answer once you had it
Opportunity = sometimes research questions also come from opportunity --> have a
neat data set? Think about what data is available to you and whether any related
research questions or theories come to mind (try to do this after understanding what is
in the data before actually analyzing the data, unless your goal is data mining)
- Something unusual or interesting happened, you might ask “what research
questions would this allow me to answer?” and from there you have a research
question, and from there a theory!
How do you know if you’ve got a good one? (Check relationship between research
question and theory)
- Consider potential results: consider the potential answers you might get -->
what kind of sense you’d make of that result or what conclusion you can draw --
> what would the results tell you about the variable? --> if you can’t say
something interesting about your potential results, that probably means your
research question and your theory aren’t as closely linked as you think
- Consider feasibility: a research question should be a question that can be
answered using the right data, if the right data is available. But is the right data
available? If answering your research question is possible but requires following
millions of people repeatedly for decades, or trying to measure something that’s
really hard to measure accurately, then that research question might not be
feasible (remember lunch 3 years ago/ getting access to private finances of
thousands unwilling people)
- Consider scale: what kind of resources and time can you dedicate to answering
the research question? --> Given the confines of, say, a term paper, you could
take some wild swings at a question, but you’re likely to do a much more
thorough job answering questions with a lot less complexity.
- Consider design: a research question can be great on its own
- Keep it simple: don’t bundle to much research questions into one
--> consider feasibility, scale, and design and keep it simple
Chapter 3 Description of variables
Description of variables: its important to accurately describe variables in empirical
research.
Variable: bunch of observations of the same measurement (in the context of empirical
research) --> successfully describing a variable = being able to take those observations
and clearly explain what was observed without making someone look through all
scores.
Types of variables: first step is to figure out what kind of variable it is
- Continuous variable: can take any value
, - Count variable: count something, how many times something has happened/
how many of something are there --> can’t be negative and can’t take fractional
values
- Ordinal variables: variables where some values are ‘more’ and others are ‘less’,
but no rule how much more is (final completed level of schooling)
- Categorical variable: recording which category an observation is in = different
options
o Special version of categorical = binary variables: categorical variables
that take 2 values (yes/no)
- Qualitative variables: catch-all category of everything else --> aren’t numeric or
categorical (text of Washington Post headline) --> tend to contain a lot of detail
that resists boiling-down-and-summarizing --> often turned into one of the other
variables above
The distribution
2. **Types of Variables**:
- **Continuous Variables**: These can take any value within a range, like income or
temperature.
- **Count Variables**: These represent counts of occurrences, like the number of
events.
- **Ordinal Variables**: These have a natural order but not a fixed interval between
them, like rankings or scales of satisfaction.
- **Categorical Variables**: These include nominal variables without intrinsic ranking,
like colors or types. Special cases include binary variables, which have only two
categories (e.g., yes/no).
- **Qualitative Variables**: These are variables that aren't numeric or categorical, like
the text of a headline.
3. **Distribution of Variables**: This section describes how to analyze and interpret the
distribution of variables:
- For categorical and ordinal variables, you can use frequency tables or bar charts to
illustrate the distribution.
- For continuous variables, distributions are typically depicted using histograms or
density plots to show the range and concentration of data values.
4. **Summarizing Distributions**: Several statistics are used to summarize
distributions, including:
