Unit 1
What to learn:
• explain what is meant with empirical research and which steps can be taken according
to the wheel of science;
• differentiate between induction and deduction;
• recognize and explain the steps that can be taken in the decision making process;
• explain the relationship between systematic decision making and systematically
answering empirical questions;
• formulate a relevant research question in the context of a specific phase in a decision
making process;
• recognize and mention examples of confirmation bias;
• explain how systematic empirical research helps avoiding the confirmation bias.
Key terms:
Research question
Design and (cycle of) decision-making
Problem & need analysis
Ex ante evaluation
Process evaluation
Ex post evaluation (effect / impact research)
Wheel of science / empirical cycle
Deduction
Induction
Confirmation bias
1. Empirical research- systematically answering empirical questions- relying on expertise of
others, empirical questions can be answered by thinking and observing at the same time.
Systematic answers are answers that cannot be answered better than other answers.
- The procedure of empirical research:
Thinking (come up with an idea)- Planning- Observing- Analysing (have conclusions and
answers to the question)
Question- Theory- Research design- Data collection- Data Analysis- Answers/Knowledge
- Starting with theory and testing theory- Deduction - we know something and then we come
up with an idea based on our knowledge
- Starting with data and trying to get conclusions based on data- Induction- we know nothing
and we make conclusions
2. Wheel of science (empirical cycle)- logical sequence of steps to defend the conclusions,
anything goes, not necessary to start with ‘theory’:
1. Think (theory)
2. Plan (research design)
3. Observe (data collection)
4. Analyse and conclude (data analysis)
,3. Normative questions- looking for justification, is this justifiable/ ‘should we…?’
4. Conceptual questions- defining theory/hypothesis
5. Empirical research questions- ‘how to’ questions (descriptive- describes a characteristic and
explanatory- cause and effect relationships) – asked in the context of decision-making
6. Decision-making- helps with answering empirical questions; problem and need analysis, we
start with designing options, evaluating options, making a choice- pick options as our
solution, implementing the choice, exposed evaluation- did our choice solve our problem?
During the decision-making we ask empirical questions
Six phases in decision-making:
- Identify, describe and analyse a problem- Problem definition and analysis
- Develop options/ design options- Design
- Compare options using criteria set in the problem definition and analysis- multi-criteria
analysis, ex ante (before) evaluation- Can we expect an option to work?’ Why?- questions’-
explanatory questions (1. Whether the option makes sense given the knowledge or theories
you have, 2. Looking at existing studies about the options you are considering, 3. Testing a
model (simulation) of what you want to evaluate- try out many things)
- Select a preferred alternative on the basis of this evaluation- decision-making rules
- Act on the decision/implement the decision- implementation (process evaluation, formative
evaluation)
- Monitor and evaluate the consequences of the selected option- ex post (outcome)
evaluation- descriptive questions
7. Problem & Need analysis- How big is the problem?/ What are its causes?/What are its
consequences?
8. Process evaluation- Did we do as planned? Did you do what you promised to do?
(implementation)- process/formative evaluation
9. Outcome evaluation- planning and implementing a new policy… did the selected option have
the expected outcome? (explanatory question)
10. Why (causes)/ what questions/ cause and effect questions- Explanatory questions
11. How many/ which questions/yes & no questions- Descriptive questions
12. Confirmation bias- search for, analyse and recall information in a way that confirms pre-
existing beliefs. It affects what we think is true and what we do. To avoid it- stick to the
procedure
- Three types of confirmation bias:
, Information (look for info that matches our beliefs)- Reasoning (we ignore and stress things
depending on what we agree with)- Conclusions (drawing conclusions depending on our
emotional issues, convictions. etc.)
- Why is confirmation bias strong?- Limitations, wishful thinking, consistency (gap between
current and new evidence)
- Consequences- mistakes in knowledge, bad decisions
- Avoiding confirmation bias- systematic data gathering- being aware of what can go wrong,
have reasoning- clarifying pre-existing beliefs, clarifying the procedure you will use to test
these beliefs (methods), sticking to these procedures
- Conspiracy ‘theory’- has consequences, type of argumentation- people look for
confirmations, disregard alternative explanations, contradict counter arguments using the
idea of ‘conspiracies’
, Unit 2
After this unit, you will be able to...
• identify units, variables (with their attributes / values) and settings in clearly formulated
descriptive and explanatory empirical research question;
• distinguish empirical questions from normative questions and from questions about
concepts;
• differentiate between explanatory empirical questions from descriptive empirical
questions.
Key terms:
Research question
Normative
Conceptual
Empirical
Explanatory
Descriptive
Unit (of analysis)
Variable (attributes / values)
Setting
Good empirical research questions have clear units and variables:
1. Units of Analysis- the objects the research question is about (people, countries,
organizations)
- Identifying unit- If the variable is known, ask ‘what or who is characterized by this
variable?’
2. Variables- possible characteristics (attributes) of these units, they describe the units,
attributes are the categories of a variable. Example: variable-job, attributes- HR manager,
truck driver, butcher, etc.
3.
- Identifying the variable- If the unit is known, ask ‘what characteristic does the unit
have?’
- Independent variable (cause)- manipulated, apply different levels of the independent
variable (ex.: the amount of water…)
- Dependent variable (effect)- measured, measure the effect on the dependent variable
(ex.: has an effect on the size of the plant)
4. Types of research questions:
- Normative (what is correct/justifiable/what should be the case?)- no observations
- Conceptual (what does it mean/ what is the meaning of the word?)- no observations
- Empirical (descriptive-description, have only one variable and explanatory (why?)-causes
and effect) (what is/will be and why?) - observations
What to learn:
• explain what is meant with empirical research and which steps can be taken according
to the wheel of science;
• differentiate between induction and deduction;
• recognize and explain the steps that can be taken in the decision making process;
• explain the relationship between systematic decision making and systematically
answering empirical questions;
• formulate a relevant research question in the context of a specific phase in a decision
making process;
• recognize and mention examples of confirmation bias;
• explain how systematic empirical research helps avoiding the confirmation bias.
Key terms:
Research question
Design and (cycle of) decision-making
Problem & need analysis
Ex ante evaluation
Process evaluation
Ex post evaluation (effect / impact research)
Wheel of science / empirical cycle
Deduction
Induction
Confirmation bias
1. Empirical research- systematically answering empirical questions- relying on expertise of
others, empirical questions can be answered by thinking and observing at the same time.
Systematic answers are answers that cannot be answered better than other answers.
- The procedure of empirical research:
Thinking (come up with an idea)- Planning- Observing- Analysing (have conclusions and
answers to the question)
Question- Theory- Research design- Data collection- Data Analysis- Answers/Knowledge
- Starting with theory and testing theory- Deduction - we know something and then we come
up with an idea based on our knowledge
- Starting with data and trying to get conclusions based on data- Induction- we know nothing
and we make conclusions
2. Wheel of science (empirical cycle)- logical sequence of steps to defend the conclusions,
anything goes, not necessary to start with ‘theory’:
1. Think (theory)
2. Plan (research design)
3. Observe (data collection)
4. Analyse and conclude (data analysis)
,3. Normative questions- looking for justification, is this justifiable/ ‘should we…?’
4. Conceptual questions- defining theory/hypothesis
5. Empirical research questions- ‘how to’ questions (descriptive- describes a characteristic and
explanatory- cause and effect relationships) – asked in the context of decision-making
6. Decision-making- helps with answering empirical questions; problem and need analysis, we
start with designing options, evaluating options, making a choice- pick options as our
solution, implementing the choice, exposed evaluation- did our choice solve our problem?
During the decision-making we ask empirical questions
Six phases in decision-making:
- Identify, describe and analyse a problem- Problem definition and analysis
- Develop options/ design options- Design
- Compare options using criteria set in the problem definition and analysis- multi-criteria
analysis, ex ante (before) evaluation- Can we expect an option to work?’ Why?- questions’-
explanatory questions (1. Whether the option makes sense given the knowledge or theories
you have, 2. Looking at existing studies about the options you are considering, 3. Testing a
model (simulation) of what you want to evaluate- try out many things)
- Select a preferred alternative on the basis of this evaluation- decision-making rules
- Act on the decision/implement the decision- implementation (process evaluation, formative
evaluation)
- Monitor and evaluate the consequences of the selected option- ex post (outcome)
evaluation- descriptive questions
7. Problem & Need analysis- How big is the problem?/ What are its causes?/What are its
consequences?
8. Process evaluation- Did we do as planned? Did you do what you promised to do?
(implementation)- process/formative evaluation
9. Outcome evaluation- planning and implementing a new policy… did the selected option have
the expected outcome? (explanatory question)
10. Why (causes)/ what questions/ cause and effect questions- Explanatory questions
11. How many/ which questions/yes & no questions- Descriptive questions
12. Confirmation bias- search for, analyse and recall information in a way that confirms pre-
existing beliefs. It affects what we think is true and what we do. To avoid it- stick to the
procedure
- Three types of confirmation bias:
, Information (look for info that matches our beliefs)- Reasoning (we ignore and stress things
depending on what we agree with)- Conclusions (drawing conclusions depending on our
emotional issues, convictions. etc.)
- Why is confirmation bias strong?- Limitations, wishful thinking, consistency (gap between
current and new evidence)
- Consequences- mistakes in knowledge, bad decisions
- Avoiding confirmation bias- systematic data gathering- being aware of what can go wrong,
have reasoning- clarifying pre-existing beliefs, clarifying the procedure you will use to test
these beliefs (methods), sticking to these procedures
- Conspiracy ‘theory’- has consequences, type of argumentation- people look for
confirmations, disregard alternative explanations, contradict counter arguments using the
idea of ‘conspiracies’
, Unit 2
After this unit, you will be able to...
• identify units, variables (with their attributes / values) and settings in clearly formulated
descriptive and explanatory empirical research question;
• distinguish empirical questions from normative questions and from questions about
concepts;
• differentiate between explanatory empirical questions from descriptive empirical
questions.
Key terms:
Research question
Normative
Conceptual
Empirical
Explanatory
Descriptive
Unit (of analysis)
Variable (attributes / values)
Setting
Good empirical research questions have clear units and variables:
1. Units of Analysis- the objects the research question is about (people, countries,
organizations)
- Identifying unit- If the variable is known, ask ‘what or who is characterized by this
variable?’
2. Variables- possible characteristics (attributes) of these units, they describe the units,
attributes are the categories of a variable. Example: variable-job, attributes- HR manager,
truck driver, butcher, etc.
3.
- Identifying the variable- If the unit is known, ask ‘what characteristic does the unit
have?’
- Independent variable (cause)- manipulated, apply different levels of the independent
variable (ex.: the amount of water…)
- Dependent variable (effect)- measured, measure the effect on the dependent variable
(ex.: has an effect on the size of the plant)
4. Types of research questions:
- Normative (what is correct/justifiable/what should be the case?)- no observations
- Conceptual (what does it mean/ what is the meaning of the word?)- no observations
- Empirical (descriptive-description, have only one variable and explanatory (why?)-causes
and effect) (what is/will be and why?) - observations
