Evidence-Based Management Reading Summary
Week 1: 1
Chapter 9: Evidence-based management, the basic principles 1
Tutorial notes: 4
Week 2: 6
Wenzel et al.: “The Double Edged Sword of Big Data in Organizational and Management Research:
A Review of Opportunities and Risks” 6
Gill et al.: “A Brief Overview of AI governance for responsible Machine Learning Systems” 16
Mitchell: “The Pillars of a Rights-Based Approach to AI Development” 19
Tutorial notes: 22
WEEK 3: 23
Rousseau: “Making evidence-based organizational decisions in an uncertain world” 23
Tingling & Brydon: “Is Decision-based evidence making necessarily bad?” 29
Week 4: 31
Merendino et al. “Big data, big decisions: the impact of big data on board level decision-making” 31
Tabesh et al. “Implementing big data strategies: A managerial perspective” 34
,Week 1:
Chapter 9: Evidence-based management, the basic principles
- Disregarding sound evidence and relying on personal experience or popular ideas is daily practice
● Evidence-based practice
= about making decisions through the conscientious, explicit and judicious use of the best
available evidence from multiple sources by__ to increase the likelihood of a favorable outcome:
➔ Asking: translating a practical issue or problem into an answerable question
➔ Acquiring: systematically searching for and retrieving the evidence
➔ Appraising: critically judging the trustworthiness and relevance of the evidence
➔ Aggregating: weighing and pulling together the evidence
➔ Applying: incorporating the evidence into the decision-making process
➔ Assessing: evaluating the outcome of the decision taken
- Good-quality decisions should be based on a combination of critical thinking and the best
available evidence
- The quality of the evidence needs to be taken into account
- Seeks to improve the way decisions are made
- An approach to decision-making and day-to-day work practice that helps practitioners
critically evaluate the extent to which they can trust the evidence they have at hand
- Helps practitioners identify, find and evaluate additional evidence relevant to their
decisions
★ Why are they needed?
- Most management decisions are not based on the best available evidence, but
rather on personal experience even though personal judgment is not a reliable
source of evidence because it is susceptible to systematic errors
- Benchmarking can demonstrate alternative way of doing things but is also not a
good indicator in itself of what would work in a different setting
★ Barriers:
- Few practitioners have been trained in the skills required to critically evaluate the
trustworthiness and relevance of information they use
- Important organizational information may be difficult to access
- The information that is available can be of poor quality
- Practitioners are often not aware of the current scientific evidence available on
key issues in the field
- Most practitioners pay little or no attention to scientific or organizational
evidence
● Evidence
- Information
- Based on numbers or can also be qualitative/ descriptive
, - Can come from scientific research (suggests generally applicable facts)
- Can come from local organizational or business indicators
- Professional experience can be an important source of evidence
- Presented in a variety of forms
- All evidence can be included if it is judged to be trustworthy and relevant
4 types of evidence:
1. Scientific evidence
- Published in academic journals
- Relevant research can also be found outside the management discipline
- It helps confirm judgments about associations, causal relationships and consequences
2. Organizational evidence
- Can be financial data, business measures, come from customers (customer satisfaction),
through employees (job satisfaction), hard numbers, but also soft elements
- Essential in identifying problems that require management’s attention
- Essential tod determine likely causes, plausible solutions and what is needed to
implement these solutions
3. Experiential evidence
- The professional experience and of judgment of managers, consultants, business leaders
and other practitioners
- Different from opinion/ belief because it is accumulated over time through reflection on
the outcomes of similar actions taken in similar situations
- Tacit knowledge
- Reflects the specialized knowledge acquired by repeated experience and practice of
specialized activities
- Can be vital for determining whether a management issue really requires attention, if
particular organizational data are trustworthy, whether research findings apply in a
particular situation
4. Stakeholder evidence
- Stakeholder value and concerns
- Reflect what stakeholders believe to be important which affects how they tend to react to
the possible consequences of the organization’s decisions
- Gathering this evidence is not just important for ethical reasons but also provides a frame
of reference from which to analyze evidence from other sources
- Provides important information about the way in which decisions will be received and
whether the outcomes of those decisions are likely to be successful
★ Sources of evidence that should be considered
- Start by asking: what is the available evidence? ⇒ what is known?
- Evidence from four sources should be taken into account:
1. Scientific research outcomes
, 2. Professional experience and judgment
3. Organizational data, facts and figures
4. Shareholders’ values and concerns
● Critically appraising evidence:
- Evidence is never perfect and can be misleading because the evidence can be
overstated or biased
- Evidence appraisal includes:
1. Finding the source of where the evidence is gathered
2. Checking if it is the best available evidence
3. Checking if the evidence provided is sufficient to come to a conclusion
4. Evaluating whether the evidence is biased
5. Assessing whether there is a reason for the evidence being biased
6. Assessing how the figures from evidence were calculated: Accuracy,
reliability; assessing how the questions and the study for gathering the
evidence were designed, what the process for data gathering was
● Focusing on best available evidence:
- The quality of our decisions is likely to improve the more use we make of
trustworthy evidence
- The best available evidence will help us make the best decisions for our current
situation
● Common misconceptions of evidence-based practice:
1. Evidence-based practice ignores the practitioner’s professional experience
- The decisions should be made through conscientious, explicit and
judicious use of evidence from the four aforementioned sources
- Evidence-based practice does not mean that any one source of evidence
is more valid than any other
2. Evidence-based practice is all about numbers and statistics
- Evidence-based practice actually seeks to use the best available evidence
from multiple sources
- Statistical thinking is an important element, but it is not what it is all
about
3. Managers need to make decisions quickly and don’t have time for
evidence-based practice
- Evidence-based practice is more often about preparing yourself and your
organization to make decisions well
- Even split-second decisions require trustworthy evidence
4. Each organization is unique, so the usefulness of scientific evidence is limited
Week 1: 1
Chapter 9: Evidence-based management, the basic principles 1
Tutorial notes: 4
Week 2: 6
Wenzel et al.: “The Double Edged Sword of Big Data in Organizational and Management Research:
A Review of Opportunities and Risks” 6
Gill et al.: “A Brief Overview of AI governance for responsible Machine Learning Systems” 16
Mitchell: “The Pillars of a Rights-Based Approach to AI Development” 19
Tutorial notes: 22
WEEK 3: 23
Rousseau: “Making evidence-based organizational decisions in an uncertain world” 23
Tingling & Brydon: “Is Decision-based evidence making necessarily bad?” 29
Week 4: 31
Merendino et al. “Big data, big decisions: the impact of big data on board level decision-making” 31
Tabesh et al. “Implementing big data strategies: A managerial perspective” 34
,Week 1:
Chapter 9: Evidence-based management, the basic principles
- Disregarding sound evidence and relying on personal experience or popular ideas is daily practice
● Evidence-based practice
= about making decisions through the conscientious, explicit and judicious use of the best
available evidence from multiple sources by__ to increase the likelihood of a favorable outcome:
➔ Asking: translating a practical issue or problem into an answerable question
➔ Acquiring: systematically searching for and retrieving the evidence
➔ Appraising: critically judging the trustworthiness and relevance of the evidence
➔ Aggregating: weighing and pulling together the evidence
➔ Applying: incorporating the evidence into the decision-making process
➔ Assessing: evaluating the outcome of the decision taken
- Good-quality decisions should be based on a combination of critical thinking and the best
available evidence
- The quality of the evidence needs to be taken into account
- Seeks to improve the way decisions are made
- An approach to decision-making and day-to-day work practice that helps practitioners
critically evaluate the extent to which they can trust the evidence they have at hand
- Helps practitioners identify, find and evaluate additional evidence relevant to their
decisions
★ Why are they needed?
- Most management decisions are not based on the best available evidence, but
rather on personal experience even though personal judgment is not a reliable
source of evidence because it is susceptible to systematic errors
- Benchmarking can demonstrate alternative way of doing things but is also not a
good indicator in itself of what would work in a different setting
★ Barriers:
- Few practitioners have been trained in the skills required to critically evaluate the
trustworthiness and relevance of information they use
- Important organizational information may be difficult to access
- The information that is available can be of poor quality
- Practitioners are often not aware of the current scientific evidence available on
key issues in the field
- Most practitioners pay little or no attention to scientific or organizational
evidence
● Evidence
- Information
- Based on numbers or can also be qualitative/ descriptive
, - Can come from scientific research (suggests generally applicable facts)
- Can come from local organizational or business indicators
- Professional experience can be an important source of evidence
- Presented in a variety of forms
- All evidence can be included if it is judged to be trustworthy and relevant
4 types of evidence:
1. Scientific evidence
- Published in academic journals
- Relevant research can also be found outside the management discipline
- It helps confirm judgments about associations, causal relationships and consequences
2. Organizational evidence
- Can be financial data, business measures, come from customers (customer satisfaction),
through employees (job satisfaction), hard numbers, but also soft elements
- Essential in identifying problems that require management’s attention
- Essential tod determine likely causes, plausible solutions and what is needed to
implement these solutions
3. Experiential evidence
- The professional experience and of judgment of managers, consultants, business leaders
and other practitioners
- Different from opinion/ belief because it is accumulated over time through reflection on
the outcomes of similar actions taken in similar situations
- Tacit knowledge
- Reflects the specialized knowledge acquired by repeated experience and practice of
specialized activities
- Can be vital for determining whether a management issue really requires attention, if
particular organizational data are trustworthy, whether research findings apply in a
particular situation
4. Stakeholder evidence
- Stakeholder value and concerns
- Reflect what stakeholders believe to be important which affects how they tend to react to
the possible consequences of the organization’s decisions
- Gathering this evidence is not just important for ethical reasons but also provides a frame
of reference from which to analyze evidence from other sources
- Provides important information about the way in which decisions will be received and
whether the outcomes of those decisions are likely to be successful
★ Sources of evidence that should be considered
- Start by asking: what is the available evidence? ⇒ what is known?
- Evidence from four sources should be taken into account:
1. Scientific research outcomes
, 2. Professional experience and judgment
3. Organizational data, facts and figures
4. Shareholders’ values and concerns
● Critically appraising evidence:
- Evidence is never perfect and can be misleading because the evidence can be
overstated or biased
- Evidence appraisal includes:
1. Finding the source of where the evidence is gathered
2. Checking if it is the best available evidence
3. Checking if the evidence provided is sufficient to come to a conclusion
4. Evaluating whether the evidence is biased
5. Assessing whether there is a reason for the evidence being biased
6. Assessing how the figures from evidence were calculated: Accuracy,
reliability; assessing how the questions and the study for gathering the
evidence were designed, what the process for data gathering was
● Focusing on best available evidence:
- The quality of our decisions is likely to improve the more use we make of
trustworthy evidence
- The best available evidence will help us make the best decisions for our current
situation
● Common misconceptions of evidence-based practice:
1. Evidence-based practice ignores the practitioner’s professional experience
- The decisions should be made through conscientious, explicit and
judicious use of evidence from the four aforementioned sources
- Evidence-based practice does not mean that any one source of evidence
is more valid than any other
2. Evidence-based practice is all about numbers and statistics
- Evidence-based practice actually seeks to use the best available evidence
from multiple sources
- Statistical thinking is an important element, but it is not what it is all
about
3. Managers need to make decisions quickly and don’t have time for
evidence-based practice
- Evidence-based practice is more often about preparing yourself and your
organization to make decisions well
- Even split-second decisions require trustworthy evidence
4. Each organization is unique, so the usefulness of scientific evidence is limited
