Learning objectives
1. How to pose a (meaningful) research question
A research question is simply a question you want to answer through research. It’s a
question about how the world works.
For quantitative empirical research, a meaningful research question must meet two key
criteria:
1. The Question Must Be Answerable
• It must be possible to imagine evidence that could credibly answer the question.
• Example: “What is the best James Bond movie?” is not answerable (too
subjective).
“Which Bond era had the highest ticket sales?” is answerable using data.
• You must also consider:
§ Feasibility: Is the required data available and measurable?
§ Scale: Is it doable with your time and resources?
§ Design: Is a realistic research setup imaginable to answer it?
2. The Answer Must Inform Theory
• The answer should help improve a broader theory — a general explanation of how
or why things happen.
• A good research question helps refine the “why” behind relationships or events.
• It often leads to a testable hypothesis.
• Test: If the result surprises you, would it force you to rethink your theory? If not,
the question might not be strongly tied to any theory.
• It should answer the “so what?” — why is the answer worth knowing?
Where Do Research Questions Come From?
• Often from curiosity about how the world works.
• Can begin with a theory: “I think this is how the world works” → leads to a
hypothesis and then a question.
• Can begin with a question: “Would this happen?” → leads to thinking about why
(theory).
• Can arise from opportunity: e.g., access to a new dataset or an unusual event.
Why Start with a Question (and Theory) Instead of Just Data Mining?
• Data mining (searching for patterns in data without a theory) is good for:
, • Finding correlations or predictions.
• But not great for improving understanding or explaining why something happens.
• It increases the risk of false positives if not done carefully.
• However, data mining can still inspire good research questions to explore further
with proper design.
How Do You Know If Your Question Is a Good One?
• Consider possible outcomes and what each would say about your theory.
• Consider feasibility (is the data available and usable?).
• Consider scope (can you answer it within your constraints?).
• Consider design (is there a realistic way to investigate it?).
• Keep it simple: Avoid bundling multiple questions into one. Break broad ideas
into smaller, focused questions.
2. How to safeguard against validity and reliability concerns
• Validity = Accuracy. Are we measuring what we intend to measure?
• Reliability = Precision. Do our measurements produce stable, consistent,
repeatable results?
Safeguarding against validity and reliability concerns focuses primarily on having a
carefully designed research design. A good design helps to:
1. Outlining as accurately as possible the process that generates the data.
2. Identifying alternative explanations for the data that do not answer the research
question.
3. Finding ways to exclude those alternative reasons in order to reveal the variation
needed.
Quantitative methods provide a "toolbox" for this and help minimize cognitive
assumptions that may bias our interpretation. Specific threats to validity are countered
by addressing them in the design, for example, by controlling for omitted variables or by
drawing a representative randomized sample. Reliability concerns are addressed by
assessing the consistency of measurements with appropriate methods.
3. Validity: Internal, external, and construct validity
Validity refers to how accurately you're measuring what you intend to measure—it's
about the accuracy of the measurement or conclusion. There are three main types of
validity discussed in the sources:
1. How to pose a (meaningful) research question
A research question is simply a question you want to answer through research. It’s a
question about how the world works.
For quantitative empirical research, a meaningful research question must meet two key
criteria:
1. The Question Must Be Answerable
• It must be possible to imagine evidence that could credibly answer the question.
• Example: “What is the best James Bond movie?” is not answerable (too
subjective).
“Which Bond era had the highest ticket sales?” is answerable using data.
• You must also consider:
§ Feasibility: Is the required data available and measurable?
§ Scale: Is it doable with your time and resources?
§ Design: Is a realistic research setup imaginable to answer it?
2. The Answer Must Inform Theory
• The answer should help improve a broader theory — a general explanation of how
or why things happen.
• A good research question helps refine the “why” behind relationships or events.
• It often leads to a testable hypothesis.
• Test: If the result surprises you, would it force you to rethink your theory? If not,
the question might not be strongly tied to any theory.
• It should answer the “so what?” — why is the answer worth knowing?
Where Do Research Questions Come From?
• Often from curiosity about how the world works.
• Can begin with a theory: “I think this is how the world works” → leads to a
hypothesis and then a question.
• Can begin with a question: “Would this happen?” → leads to thinking about why
(theory).
• Can arise from opportunity: e.g., access to a new dataset or an unusual event.
Why Start with a Question (and Theory) Instead of Just Data Mining?
• Data mining (searching for patterns in data without a theory) is good for:
, • Finding correlations or predictions.
• But not great for improving understanding or explaining why something happens.
• It increases the risk of false positives if not done carefully.
• However, data mining can still inspire good research questions to explore further
with proper design.
How Do You Know If Your Question Is a Good One?
• Consider possible outcomes and what each would say about your theory.
• Consider feasibility (is the data available and usable?).
• Consider scope (can you answer it within your constraints?).
• Consider design (is there a realistic way to investigate it?).
• Keep it simple: Avoid bundling multiple questions into one. Break broad ideas
into smaller, focused questions.
2. How to safeguard against validity and reliability concerns
• Validity = Accuracy. Are we measuring what we intend to measure?
• Reliability = Precision. Do our measurements produce stable, consistent,
repeatable results?
Safeguarding against validity and reliability concerns focuses primarily on having a
carefully designed research design. A good design helps to:
1. Outlining as accurately as possible the process that generates the data.
2. Identifying alternative explanations for the data that do not answer the research
question.
3. Finding ways to exclude those alternative reasons in order to reveal the variation
needed.
Quantitative methods provide a "toolbox" for this and help minimize cognitive
assumptions that may bias our interpretation. Specific threats to validity are countered
by addressing them in the design, for example, by controlling for omitted variables or by
drawing a representative randomized sample. Reliability concerns are addressed by
assessing the consistency of measurements with appropriate methods.
3. Validity: Internal, external, and construct validity
Validity refers to how accurately you're measuring what you intend to measure—it's
about the accuracy of the measurement or conclusion. There are three main types of
validity discussed in the sources:
