DBR
SUMMARY
Sarah van Notten
ACADEMIC YEAR 2021 - 2022 | BLOCK A
,SARAH VAN NOTTEN DBR | ACADEMIC YEAR 2021 – 2022 | BLOCK A
TABLE OF CONTENTS
1. The Basic Principles of Evidence-Based Management ..................................................... 4
1.1 What is evidence-based management? ................................................................................. 4
1.2 What counts as evidence? ...................................................................................................... 4
1.3 Why do we need evidence-based management? .................................................................. 4
1.4 What sources of evidence should be considered? ................................................................. 5
1.5 Why do we have to critically appraise evidence? .................................................................. 5
1.6 Why focus on the best available evidence? ........................................................................... 5
1.7 What is the evidence for evidence-based management? ...................................................... 6
1.8 SUMMARY............................................................................................................................... 6
2. ASK critical questions about problems and solutions ...................................................... 7
2.1 Identifying underlying assumptions ....................................................................................... 7
2.2 Starting point: what is the problem to solve? & PICOC ......................................................... 7
2.3 Overview of Questions ........................................................................................................... 8
2.4 SUMMARY............................................................................................................................... 9
3. ACQUIRE Evidence from Practitioners .......................................................................... 11
3.1 What to Ask .......................................................................................................................... 11
3.2 Whom to Ask ........................................................................................................................ 11
3.3 How to Ask ............................................................................................................................ 12
3.4 Developing Questions ........................................................................................................... 12
3.5 SUMMARY............................................................................................................................. 13
4. APPRAISE Evidence from Practitioners ......................................................................... 14
4.1 What constitutes valid and reliable professional expertise? ............................................... 14
4.2 Two modes of thinking ......................................................................................................... 14
4.3 Cognitive biases .................................................................................................................... 15
4.4 Patternicity and the illusion of causality .............................................................................. 15
4.5 Confirmation bias ................................................................................................................. 16
4.6 Group conformity ................................................................................................................. 16
4.7 Other common cognitive biases ........................................................................................... 16
4.8 Avoiding or reducing bias ..................................................................................................... 16
4.9 Critically appraising evidence from practitioners ................................................................. 17
4.10 Overview of Appraisal Questions ......................................................................................... 17
4.11 SUMMARY............................................................................................................................. 18
5. ACQUIRE Evidence from Scientific Literature ................................................................ 19
5.1 Where do we search? ........................................................................................................... 19
5.2 PICOC .................................................................................................................................... 20
5.3 Determining your search terms ............................................................................................ 20
5.4 Conducting your research..................................................................................................... 20
5.5 Final tips & tricks .................................................................................................................. 21
5.6 SUMMARY............................................................................................................................. 21
7. APPRAISE Evidence from Scientific Literature ............................................................... 22
7.1 Question 1: Are the findings of the study of practical relevance? ....................................... 22
7.2 Question 2: How precise are the findings? .......................................................................... 23
7.3 Question 3: How were the findings measured? ................................................................... 23
7.4 Critical appraisal: methodological appropriateness ............................................................. 23
7.5 Methodological appropriateness: cause-and-effect studies................................................ 23
7.6 Methodological appropriateness: non-effect studies .......................................................... 25
2
,SARAH VAN NOTTEN DBR | ACADEMIC YEAR 2021 – 2022 | BLOCK A
7.7 Methodological appropriateness: summary ........................................................................ 26
7.8 Critical appraisal: methodology quality ................................................................................ 29
7.9 Methodological quality: weaknesses ................................................................................... 29
7.10 Finally: The best available evidence ..................................................................................... 31
7.11 SUMMARY............................................................................................................................. 32
8. ACQUIRE evidence from the organization (beta) .......................................................... 33
8.1 Why is evidence from the organization important? ............................................................ 33
8.2 From data to information to evidence ................................................................................. 33
8.3 What questions to ask? ........................................................................................................ 34
8.4 What types of evidence are typically available? .................................................................. 34
8.5 Big data ................................................................................................................................. 35
8.6 Where to find organizational evidence ................................................................................ 35
8.7 Techniques and tools for acquiring data .............................................................................. 36
8.8 Organizational data turned into information ....................................................................... 36
8.9 Some final considerations when acquiring data .................................................................. 37
8.10 SUMMARY............................................................................................................................. 38
9. APPRAISE evidence from the organization ................................................................... 39
9.1 Barrier 1: Absence of a logic model ...................................................................................... 39
9.2 Barrier 2: Irrelevant data ...................................................................................................... 39
9.3 Barrier 3: Inaccurate data ..................................................................................................... 39
9.4 Barrier 4: Missing contextual information ........................................................................... 40
9.5 Barrier 5: Measurement error .............................................................................................. 40
9.6 Barrier 6: The small number problem .................................................................................. 40
9.7 Barrier 7: confusing percentages and averages ................................................................... 41
9.8 Barrier 8: misleading graphs ................................................................................................. 41
9.9 Barrier 9: correlations & regressions – overfitting the data ................................................ 41
9.10 Barrier 10: wide confidence intervals ................................................................................... 42
9.11 Overview of appraisal questions .......................................................................................... 42
9.12 Big data, machine learning, and artificial neural networks .................................................. 43
9.13 Finally: organizational evidence – quality versus purpose ................................................... 43
9.14 SUMMARY............................................................................................................................. 43
13. APPLY: incorporate evidence into your decision ......................................................... 45
13.1 Consideration 1: does the evidence apply? ......................................................................... 45
13.2 Consideration 2: what is the expected value? ..................................................................... 45
13.3 Consideration 3: is it the biggest bang for your buck? ......................................................... 46
13.4 Consideration 4: is the level of risk acceptable? .................................................................. 46
13.5 Consideration 5: are there ethical issues to consider? ........................................................ 46
13.6 Consideration 6: is the evidence actionable? ....................................................................... 46
13.7 Consideration 7: are there moderators to take into account? ............................................ 46
13.8 How and in what form can you apply the evidence? ........................................................... 46
13.9 SUMMARY............................................................................................................................. 47
14. ASSESS: evaluate the outcome of the decision ............................................................ 49
14.1 Types of decisions ................................................................................................................. 49
14.2 Assessing the outcome: two preliminary questions ............................................................ 49
14.3 Assess: the gold standard ..................................................................................................... 50
14.4 Assess: the silver standard ................................................................................................... 50
14.5 Outcome measures............................................................................................................... 51
14.6 Assessing stakeholders effects ............................................................................................. 51
14.7 SUMMARY............................................................................................................................. 52
3
, SARAH VAN NOTTEN DBR | ACADEMIC YEAR 2021 – 2022 | BLOCK A
1. The Basic Principles of Evidence-Based Management
1.1 What is evidence-based management?
Evidence-based management is about making decisions through the conscientious, explicit, judicious use of
the best available evidence from multiple sources by:
• Asking: translating a practical issue or problem into an answerable question.
• Acquiring: systematically searching for and retrieving the evidence.
• Appraising: critically judging the trustworthiness and relevance of the evidence.
• Aggregating: weighing and pulling together the evidence.
• Applying: incorporating the evidence into the decision-making process.
• Assessing: evaluating the outcome of the decision taken.
To increase the likelihood of a favorable outcome.
Three most important elements of evidence-based management:
Evidence-based management is about…
• taking a systematic (six-step) approach to decision-making.
• basing your decisions on multiple sources of evidence.
• basing your decisions on critically appraised evidence.
1.2 What counts as evidence?
Evidence = information (facts or date supporting/contradicting a claim, assumption, or hypothesis).
Evidence can be based on numbers, or it can be qualitative or descriptive. It can come from scientific research.
It can come from local organizational or business indicators. It can come from professional experience.
1.3 Why do we need evidence-based management?
Most management decisions are not based on best available evidence:
• Practitioners base decisions solely on their judgment or personal experience.
• Personal experience alone is not a reliable source.
• Common source of evidence: benchmarking and best practices benchmarking can demonstrate
alternative practices but is not a good indicator of what would work in a different setting.
Barriers to evidence-based practice:
• Few practitioners have been trained in the skills required to critically evaluate the information.
• Important organizational information may be difficult to access.
4
SUMMARY
Sarah van Notten
ACADEMIC YEAR 2021 - 2022 | BLOCK A
,SARAH VAN NOTTEN DBR | ACADEMIC YEAR 2021 – 2022 | BLOCK A
TABLE OF CONTENTS
1. The Basic Principles of Evidence-Based Management ..................................................... 4
1.1 What is evidence-based management? ................................................................................. 4
1.2 What counts as evidence? ...................................................................................................... 4
1.3 Why do we need evidence-based management? .................................................................. 4
1.4 What sources of evidence should be considered? ................................................................. 5
1.5 Why do we have to critically appraise evidence? .................................................................. 5
1.6 Why focus on the best available evidence? ........................................................................... 5
1.7 What is the evidence for evidence-based management? ...................................................... 6
1.8 SUMMARY............................................................................................................................... 6
2. ASK critical questions about problems and solutions ...................................................... 7
2.1 Identifying underlying assumptions ....................................................................................... 7
2.2 Starting point: what is the problem to solve? & PICOC ......................................................... 7
2.3 Overview of Questions ........................................................................................................... 8
2.4 SUMMARY............................................................................................................................... 9
3. ACQUIRE Evidence from Practitioners .......................................................................... 11
3.1 What to Ask .......................................................................................................................... 11
3.2 Whom to Ask ........................................................................................................................ 11
3.3 How to Ask ............................................................................................................................ 12
3.4 Developing Questions ........................................................................................................... 12
3.5 SUMMARY............................................................................................................................. 13
4. APPRAISE Evidence from Practitioners ......................................................................... 14
4.1 What constitutes valid and reliable professional expertise? ............................................... 14
4.2 Two modes of thinking ......................................................................................................... 14
4.3 Cognitive biases .................................................................................................................... 15
4.4 Patternicity and the illusion of causality .............................................................................. 15
4.5 Confirmation bias ................................................................................................................. 16
4.6 Group conformity ................................................................................................................. 16
4.7 Other common cognitive biases ........................................................................................... 16
4.8 Avoiding or reducing bias ..................................................................................................... 16
4.9 Critically appraising evidence from practitioners ................................................................. 17
4.10 Overview of Appraisal Questions ......................................................................................... 17
4.11 SUMMARY............................................................................................................................. 18
5. ACQUIRE Evidence from Scientific Literature ................................................................ 19
5.1 Where do we search? ........................................................................................................... 19
5.2 PICOC .................................................................................................................................... 20
5.3 Determining your search terms ............................................................................................ 20
5.4 Conducting your research..................................................................................................... 20
5.5 Final tips & tricks .................................................................................................................. 21
5.6 SUMMARY............................................................................................................................. 21
7. APPRAISE Evidence from Scientific Literature ............................................................... 22
7.1 Question 1: Are the findings of the study of practical relevance? ....................................... 22
7.2 Question 2: How precise are the findings? .......................................................................... 23
7.3 Question 3: How were the findings measured? ................................................................... 23
7.4 Critical appraisal: methodological appropriateness ............................................................. 23
7.5 Methodological appropriateness: cause-and-effect studies................................................ 23
7.6 Methodological appropriateness: non-effect studies .......................................................... 25
2
,SARAH VAN NOTTEN DBR | ACADEMIC YEAR 2021 – 2022 | BLOCK A
7.7 Methodological appropriateness: summary ........................................................................ 26
7.8 Critical appraisal: methodology quality ................................................................................ 29
7.9 Methodological quality: weaknesses ................................................................................... 29
7.10 Finally: The best available evidence ..................................................................................... 31
7.11 SUMMARY............................................................................................................................. 32
8. ACQUIRE evidence from the organization (beta) .......................................................... 33
8.1 Why is evidence from the organization important? ............................................................ 33
8.2 From data to information to evidence ................................................................................. 33
8.3 What questions to ask? ........................................................................................................ 34
8.4 What types of evidence are typically available? .................................................................. 34
8.5 Big data ................................................................................................................................. 35
8.6 Where to find organizational evidence ................................................................................ 35
8.7 Techniques and tools for acquiring data .............................................................................. 36
8.8 Organizational data turned into information ....................................................................... 36
8.9 Some final considerations when acquiring data .................................................................. 37
8.10 SUMMARY............................................................................................................................. 38
9. APPRAISE evidence from the organization ................................................................... 39
9.1 Barrier 1: Absence of a logic model ...................................................................................... 39
9.2 Barrier 2: Irrelevant data ...................................................................................................... 39
9.3 Barrier 3: Inaccurate data ..................................................................................................... 39
9.4 Barrier 4: Missing contextual information ........................................................................... 40
9.5 Barrier 5: Measurement error .............................................................................................. 40
9.6 Barrier 6: The small number problem .................................................................................. 40
9.7 Barrier 7: confusing percentages and averages ................................................................... 41
9.8 Barrier 8: misleading graphs ................................................................................................. 41
9.9 Barrier 9: correlations & regressions – overfitting the data ................................................ 41
9.10 Barrier 10: wide confidence intervals ................................................................................... 42
9.11 Overview of appraisal questions .......................................................................................... 42
9.12 Big data, machine learning, and artificial neural networks .................................................. 43
9.13 Finally: organizational evidence – quality versus purpose ................................................... 43
9.14 SUMMARY............................................................................................................................. 43
13. APPLY: incorporate evidence into your decision ......................................................... 45
13.1 Consideration 1: does the evidence apply? ......................................................................... 45
13.2 Consideration 2: what is the expected value? ..................................................................... 45
13.3 Consideration 3: is it the biggest bang for your buck? ......................................................... 46
13.4 Consideration 4: is the level of risk acceptable? .................................................................. 46
13.5 Consideration 5: are there ethical issues to consider? ........................................................ 46
13.6 Consideration 6: is the evidence actionable? ....................................................................... 46
13.7 Consideration 7: are there moderators to take into account? ............................................ 46
13.8 How and in what form can you apply the evidence? ........................................................... 46
13.9 SUMMARY............................................................................................................................. 47
14. ASSESS: evaluate the outcome of the decision ............................................................ 49
14.1 Types of decisions ................................................................................................................. 49
14.2 Assessing the outcome: two preliminary questions ............................................................ 49
14.3 Assess: the gold standard ..................................................................................................... 50
14.4 Assess: the silver standard ................................................................................................... 50
14.5 Outcome measures............................................................................................................... 51
14.6 Assessing stakeholders effects ............................................................................................. 51
14.7 SUMMARY............................................................................................................................. 52
3
, SARAH VAN NOTTEN DBR | ACADEMIC YEAR 2021 – 2022 | BLOCK A
1. The Basic Principles of Evidence-Based Management
1.1 What is evidence-based management?
Evidence-based management is about making decisions through the conscientious, explicit, judicious use of
the best available evidence from multiple sources by:
• Asking: translating a practical issue or problem into an answerable question.
• Acquiring: systematically searching for and retrieving the evidence.
• Appraising: critically judging the trustworthiness and relevance of the evidence.
• Aggregating: weighing and pulling together the evidence.
• Applying: incorporating the evidence into the decision-making process.
• Assessing: evaluating the outcome of the decision taken.
To increase the likelihood of a favorable outcome.
Three most important elements of evidence-based management:
Evidence-based management is about…
• taking a systematic (six-step) approach to decision-making.
• basing your decisions on multiple sources of evidence.
• basing your decisions on critically appraised evidence.
1.2 What counts as evidence?
Evidence = information (facts or date supporting/contradicting a claim, assumption, or hypothesis).
Evidence can be based on numbers, or it can be qualitative or descriptive. It can come from scientific research.
It can come from local organizational or business indicators. It can come from professional experience.
1.3 Why do we need evidence-based management?
Most management decisions are not based on best available evidence:
• Practitioners base decisions solely on their judgment or personal experience.
• Personal experience alone is not a reliable source.
• Common source of evidence: benchmarking and best practices benchmarking can demonstrate
alternative practices but is not a good indicator of what would work in a different setting.
Barriers to evidence-based practice:
• Few practitioners have been trained in the skills required to critically evaluate the information.
• Important organizational information may be difficult to access.
4
