Summary Kettl – Little Bites of Big Data
Chapter 1 – knowing better
We can’t differentiate between what we know and what we don’t
o People grab onto what they think they know, it’s easier than confronting gaps in our
knowledge thus misinformation shapes our decisions, esp when the gap doesn’t
cause us personal concern, others’ opinions can be written off as idiotic; this isn’t on
purpose, people don’t do stupid things on purpose.
We (think we) know more than before
o Because of big data, but this has to be translated into actionable insights with
advanced tools (and that doesn’t always happen correctly)
Some evidence can be found for pretty much everything the govt does
o But evidence isn’t always right
Decision makers don’t always follow what we know
o Because they only seek evidence that’s useful for them. And their decisions don’t
always lead the country in the right direction, example: war in Iraq – based on flawed
intelligence that wasn’t challenged, led to a long war with little gains
We don’t know everything, and we never can
o The human brain is limited (we can’t understand and process everything)
Some of what we (think we) know is wrong
o Example: in the Iraq war policymakers believed the movie ‘The Rock’, in which
nuclear weapons were carried in a glass container, this was referenced in a report
pre-invasion – this isn’t true in real life, glass can break and the weapon can then
hurt the people planning to use them. Esp elected officials overestimate their ideas,
bc their ideas get reinforced by crowds of cheering fans & their votes.
We don’t need evidence to make decisions
o Policy analysts greatly overestimate the amount and distinctiveness of information
and analysis they offer for social problem solving – society can rely on “ordinary
knowledge” for decision making (experience and common sense)
Policymakers will only use evidence if it’s useful to them
o They only seek evidence for problems they need or want to solve
Thus: producing more evidence won’t affect policymaking
o If the evidence is not useful for them, they won’t use/accept it; often, they’re also
answers to questions that policymakers aren’t interested in
Policymakers often trust their own instincts more than studies
o As studies often aren’t black-and-white, they’re often backward (instead of forward)
looking, and sometimes difficult to understand (when using complex algorithms)
Making public policy better requires closing this^ gap between supply and demand in
evidence
o Five principles:
Evidence is of no use unless its consumers want it and use it
It’s important to get the story and get it right
It’s important to tell the story in ways that capture what the evidence says
(and thus fits what the analysts who supply it know), and in language that’s
clear (and thus fits what policymakers who demand it need to know)
It’s important to sell the story in ways that make the evidence convincing
It’s important that the evidence speaks above the noise
, Chapter 2 – get the story, get it right
Five fundamental questions for which policymakers need good answers
Hindsight
Foresight
Results
Risk
Resilience
Hindsight
o Trough program evaluation and randomized controlled trials. But: hard to randomly
assign individuals to treatment or control group, and it can be hard to deny people
the treatment, and they’re expensive and time-intensive.
o Policymakers don’t easily let go of ideas they believe in knowing better doesn’t
necessarily lead to doing better, even when RCT’s prove something (example:
showing kids for one day what prison is like doesn’t deter them, it increased crime,
but the program continued nonetheless)
Foresight
o Foresight is never perfect, e.g. the economic recession of 2008; computers don’t
always tell policymakers that the world has changed (and the eco crisis would be
worse/longer than expected); and policymakers don’t always ask the right questions
Results
o Policymakers need to see trends as they’re emerging (performance management –
mapping what’s happening in real-time), which is dependent on:
Accurate and timely intelligence
Rapid deployment
Effective tactics
Relentless follow-up and assessment
Risk
o Policymakers don’t always assess risks very well, or sometimes it’s impossible to do
so (example: in Flint, they decided not to install a corrosion control system which led
to high levels of lead in the water, which causes health problems). Important for risk
management:
Get the leaders on board
Identify risks
Assess risks
Develop an action plan (a response to risks)
Monitor risks
Communicate about risks
Resilience
o Public programs need resilience to prepare for disruptions, to recover from shocks
and to adapt and grow from disruptive experiences– BUT, it’s almost always cheaper
to avoid/mitigate problems than to clean them up afterwards, it’s just not always
possible to avoid/mitigate every problem
Good answers to these questions
o Three main puzzles when it comes to good answers:
Silos of information
Providing convincing answers
Dealing with values (that policymakers and citizens believe in), the prism
through which we look at evidence
Chapter 1 – knowing better
We can’t differentiate between what we know and what we don’t
o People grab onto what they think they know, it’s easier than confronting gaps in our
knowledge thus misinformation shapes our decisions, esp when the gap doesn’t
cause us personal concern, others’ opinions can be written off as idiotic; this isn’t on
purpose, people don’t do stupid things on purpose.
We (think we) know more than before
o Because of big data, but this has to be translated into actionable insights with
advanced tools (and that doesn’t always happen correctly)
Some evidence can be found for pretty much everything the govt does
o But evidence isn’t always right
Decision makers don’t always follow what we know
o Because they only seek evidence that’s useful for them. And their decisions don’t
always lead the country in the right direction, example: war in Iraq – based on flawed
intelligence that wasn’t challenged, led to a long war with little gains
We don’t know everything, and we never can
o The human brain is limited (we can’t understand and process everything)
Some of what we (think we) know is wrong
o Example: in the Iraq war policymakers believed the movie ‘The Rock’, in which
nuclear weapons were carried in a glass container, this was referenced in a report
pre-invasion – this isn’t true in real life, glass can break and the weapon can then
hurt the people planning to use them. Esp elected officials overestimate their ideas,
bc their ideas get reinforced by crowds of cheering fans & their votes.
We don’t need evidence to make decisions
o Policy analysts greatly overestimate the amount and distinctiveness of information
and analysis they offer for social problem solving – society can rely on “ordinary
knowledge” for decision making (experience and common sense)
Policymakers will only use evidence if it’s useful to them
o They only seek evidence for problems they need or want to solve
Thus: producing more evidence won’t affect policymaking
o If the evidence is not useful for them, they won’t use/accept it; often, they’re also
answers to questions that policymakers aren’t interested in
Policymakers often trust their own instincts more than studies
o As studies often aren’t black-and-white, they’re often backward (instead of forward)
looking, and sometimes difficult to understand (when using complex algorithms)
Making public policy better requires closing this^ gap between supply and demand in
evidence
o Five principles:
Evidence is of no use unless its consumers want it and use it
It’s important to get the story and get it right
It’s important to tell the story in ways that capture what the evidence says
(and thus fits what the analysts who supply it know), and in language that’s
clear (and thus fits what policymakers who demand it need to know)
It’s important to sell the story in ways that make the evidence convincing
It’s important that the evidence speaks above the noise
, Chapter 2 – get the story, get it right
Five fundamental questions for which policymakers need good answers
Hindsight
Foresight
Results
Risk
Resilience
Hindsight
o Trough program evaluation and randomized controlled trials. But: hard to randomly
assign individuals to treatment or control group, and it can be hard to deny people
the treatment, and they’re expensive and time-intensive.
o Policymakers don’t easily let go of ideas they believe in knowing better doesn’t
necessarily lead to doing better, even when RCT’s prove something (example:
showing kids for one day what prison is like doesn’t deter them, it increased crime,
but the program continued nonetheless)
Foresight
o Foresight is never perfect, e.g. the economic recession of 2008; computers don’t
always tell policymakers that the world has changed (and the eco crisis would be
worse/longer than expected); and policymakers don’t always ask the right questions
Results
o Policymakers need to see trends as they’re emerging (performance management –
mapping what’s happening in real-time), which is dependent on:
Accurate and timely intelligence
Rapid deployment
Effective tactics
Relentless follow-up and assessment
Risk
o Policymakers don’t always assess risks very well, or sometimes it’s impossible to do
so (example: in Flint, they decided not to install a corrosion control system which led
to high levels of lead in the water, which causes health problems). Important for risk
management:
Get the leaders on board
Identify risks
Assess risks
Develop an action plan (a response to risks)
Monitor risks
Communicate about risks
Resilience
o Public programs need resilience to prepare for disruptions, to recover from shocks
and to adapt and grow from disruptive experiences– BUT, it’s almost always cheaper
to avoid/mitigate problems than to clean them up afterwards, it’s just not always
possible to avoid/mitigate every problem
Good answers to these questions
o Three main puzzles when it comes to good answers:
Silos of information
Providing convincing answers
Dealing with values (that policymakers and citizens believe in), the prism
through which we look at evidence
