Index
Systematic Reviews in the Social Sciences: a practical guide – Petticrew and Roberts (2006)...............4
Chapter 1: Why do we need systematic reviews?..............................................................................4
1.1 Introduction..............................................................................................................................4
1.2 Bias in traditional reviews.........................................................................................................5
1.3 Information overload................................................................................................................5
1.4 The role of systematic literature reviews..................................................................................5
1.5 When a systematic review is of value.......................................................................................6
1.6 Why do a systematic review? A rationale.................................................................................7
1.7 An unsystematic history of systematic reviews........................................................................7
1.8 The Cochrane and Campbell collaborations: international collaborations of systematic
reviewers........................................................................................................................................7
1.9 Systematic reviews as paradigm shifters..................................................................................7
1.10 When to do a systematic review.............................................................................................7
Key learning points.........................................................................................................................8
Chapter 2: Starting the review – Refining the question and defining the boundaries........................9
2.1Is a systematic review the right tool for the job?......................................................................9
2.2 Framing the review question....................................................................................................9
2.3 Systematic reviews as hypothesis tests..................................................................................10
2.4 When should one do a review – early in the development of the field? Or later, when there
are many primary studies to review?...........................................................................................11
2.5 Start by finding existing systematic reviews...........................................................................11
2.6 Alternatives to systematic reviews.........................................................................................11
2.7 Defining the question: break the question down...................................................................12
2.8 No magic bullet.......................................................................................................................13
2.9 Review protocols....................................................................................................................13
2.10 What sorts of questions can systematic reviews answer?....................................................13
2.11 Scoping reviews....................................................................................................................14
Chapter 3: What sorts of studies do I include in the review? Deciding on the review’s
inclusion/exclusion criteria...............................................................................................................15
3.1 Hierarchies of evidence..........................................................................................................15
3.2 Setting inclusion/exclusion criteria: deciding what types of studies to include and exclude. .16
3.3 Use of qualitative research in systematic reviews..................................................................17
3.4 Inclusion criteria for outcomes...............................................................................................18
, 3.5 ‘Narrow’ versus ‘broad’ reviews.............................................................................................18
Key learning points.......................................................................................................................18
Chapter 4: How to find the studies – the literature search...............................................................19
4.2 Where to start?.......................................................................................................................19
4.3 Using electronic databases: search strategies........................................................................19
4.4 The right stuff: sensitivity and specificity................................................................................19
4.8 Citation searching...................................................................................................................19
4.9 Current awareness..................................................................................................................19
4.10 The web................................................................................................................................20
4.11 How do you know when to stop searching? Some stopping rules........................................20
4.12 How many databases should one search?............................................................................20
4.13 Keeping records....................................................................................................................20
4.14 1966 and all that: How far back should one search?............................................................21
4.15 Deciding which articles to obtain in hard copy.....................................................................21
4.16 Managing the references......................................................................................................21
4.17 Contacts with experts...........................................................................................................21
Chapter 5: How to appraise the studies: An introduction to assessing study quality.......................22
5.1 Introduction............................................................................................................................22
5.2 Non-reporting by authors: when no news isn’t always good news........................................22
5.3 What is study quality?............................................................................................................22
5.4 Critical appraisal: too much criticism, not enough appraisal?.................................................23
5.5 The hierarchy of evidence again.............................................................................................23
5.6 In search of the methodological ‘holy grail’............................................................................23
5.7 Critical appraisal: tools, scales and checklists.........................................................................24
5.8 Inside the black box: assessing the implementation of the intervention................................25
5.9 Treatment integrity, implementation, and context................................................................25
5.10 Assessing external validity....................................................................................................26
5.11 Quality assessment and qualitative research........................................................................26
5.12 Does quality assessment matter?.........................................................................................27
5.13 Data extraction.....................................................................................................................27
5.14 Checking the reliability of critical appraisal and data extraction..........................................28
5.15 Extracting data on differential effects of interventions........................................................28
5.16 Translating non-English papers.............................................................................................28
5.17 Concluding comments..........................................................................................................28
, Key learning points.......................................................................................................................28
Chapter 7: Exploring heterogeneity and publication bias.................................................................30
7.7 Graphical exploration of differences between studies: forest, funnel and l’abbé plots.........30
7.8 Publication bias.......................................................................................................................31
7.13 Publication bias and qualitative research.............................................................................33
Key learning points.......................................................................................................................33
Chapter 9: Systematic reviews: Urban myths and fairy tales............................................................33
9.1 Systematic reviews and urban myths.....................................................................................33
9.5 Conclusion..............................................................................................................................34
, Systematic Reviews in the Social Sciences: a practical guide – Petticrew
and Roberts (2006)
Chapter 1: Why do we need systematic reviews?
1.1 Introduction
Systematic literature reviews method of making sense of large bodies of information, and means of
contributing to the answers to questions about what does and doesn’t work
- Mapping out areas of uncertainty
- Identify where little/no relevant research has been done where new studies are needed
Important to be able to tell the difference between real and assumed knowledge systematic
reviews can help with this
Reviews are important because the results of single studies are generally given much greater
credence than they merit
Most research can only be understood in context key part of context consists of the results of
other studies that tested the same hypothesis, in similar populations
Box 1.2: On the importance of knowing about uncertainty
Lao-Tze: ‘Important not only that we know what we know, but that we know what we don’t
know’.
Confucius: ‘When you know a thing, to hold that you know it; and when you don’t know a thing,
to allow that you don’t know it – this is knowledge.’
Rumsfeld: ‘There are known knowns: the things we know we know. There are also known
unknowns: we know there are some things we don’t know. There are also unknown unknowns,
the ones we don’t know we don’t know.’
One problem with interpreting and using research is that the research behind headlines is often far
removed from real-life settings may be difficult for policymakers or public to know whether the
results are to be taken seriously
- The more sceptical research-informed policymaker may simply wait patiently, on the grounds that
another researcher will soon publish a paper saying the opposite
Studies produce conflicting findings for other reasons than methodological shortcomings or authorial
bias
findings may often be statistically significant, but not socially significant
Sometimes claims with strong face validity (headlines) can be attractive to politicians, and seized on
as evidence to support a call to action.
Literature reviews may themselves be biased, and by carefully selecting which studies to review, it’s
possible to produce 2 similar reviews that come to entirely opposite conclusions.
Systematic Reviews in the Social Sciences: a practical guide – Petticrew and Roberts (2006)...............4
Chapter 1: Why do we need systematic reviews?..............................................................................4
1.1 Introduction..............................................................................................................................4
1.2 Bias in traditional reviews.........................................................................................................5
1.3 Information overload................................................................................................................5
1.4 The role of systematic literature reviews..................................................................................5
1.5 When a systematic review is of value.......................................................................................6
1.6 Why do a systematic review? A rationale.................................................................................7
1.7 An unsystematic history of systematic reviews........................................................................7
1.8 The Cochrane and Campbell collaborations: international collaborations of systematic
reviewers........................................................................................................................................7
1.9 Systematic reviews as paradigm shifters..................................................................................7
1.10 When to do a systematic review.............................................................................................7
Key learning points.........................................................................................................................8
Chapter 2: Starting the review – Refining the question and defining the boundaries........................9
2.1Is a systematic review the right tool for the job?......................................................................9
2.2 Framing the review question....................................................................................................9
2.3 Systematic reviews as hypothesis tests..................................................................................10
2.4 When should one do a review – early in the development of the field? Or later, when there
are many primary studies to review?...........................................................................................11
2.5 Start by finding existing systematic reviews...........................................................................11
2.6 Alternatives to systematic reviews.........................................................................................11
2.7 Defining the question: break the question down...................................................................12
2.8 No magic bullet.......................................................................................................................13
2.9 Review protocols....................................................................................................................13
2.10 What sorts of questions can systematic reviews answer?....................................................13
2.11 Scoping reviews....................................................................................................................14
Chapter 3: What sorts of studies do I include in the review? Deciding on the review’s
inclusion/exclusion criteria...............................................................................................................15
3.1 Hierarchies of evidence..........................................................................................................15
3.2 Setting inclusion/exclusion criteria: deciding what types of studies to include and exclude. .16
3.3 Use of qualitative research in systematic reviews..................................................................17
3.4 Inclusion criteria for outcomes...............................................................................................18
, 3.5 ‘Narrow’ versus ‘broad’ reviews.............................................................................................18
Key learning points.......................................................................................................................18
Chapter 4: How to find the studies – the literature search...............................................................19
4.2 Where to start?.......................................................................................................................19
4.3 Using electronic databases: search strategies........................................................................19
4.4 The right stuff: sensitivity and specificity................................................................................19
4.8 Citation searching...................................................................................................................19
4.9 Current awareness..................................................................................................................19
4.10 The web................................................................................................................................20
4.11 How do you know when to stop searching? Some stopping rules........................................20
4.12 How many databases should one search?............................................................................20
4.13 Keeping records....................................................................................................................20
4.14 1966 and all that: How far back should one search?............................................................21
4.15 Deciding which articles to obtain in hard copy.....................................................................21
4.16 Managing the references......................................................................................................21
4.17 Contacts with experts...........................................................................................................21
Chapter 5: How to appraise the studies: An introduction to assessing study quality.......................22
5.1 Introduction............................................................................................................................22
5.2 Non-reporting by authors: when no news isn’t always good news........................................22
5.3 What is study quality?............................................................................................................22
5.4 Critical appraisal: too much criticism, not enough appraisal?.................................................23
5.5 The hierarchy of evidence again.............................................................................................23
5.6 In search of the methodological ‘holy grail’............................................................................23
5.7 Critical appraisal: tools, scales and checklists.........................................................................24
5.8 Inside the black box: assessing the implementation of the intervention................................25
5.9 Treatment integrity, implementation, and context................................................................25
5.10 Assessing external validity....................................................................................................26
5.11 Quality assessment and qualitative research........................................................................26
5.12 Does quality assessment matter?.........................................................................................27
5.13 Data extraction.....................................................................................................................27
5.14 Checking the reliability of critical appraisal and data extraction..........................................28
5.15 Extracting data on differential effects of interventions........................................................28
5.16 Translating non-English papers.............................................................................................28
5.17 Concluding comments..........................................................................................................28
, Key learning points.......................................................................................................................28
Chapter 7: Exploring heterogeneity and publication bias.................................................................30
7.7 Graphical exploration of differences between studies: forest, funnel and l’abbé plots.........30
7.8 Publication bias.......................................................................................................................31
7.13 Publication bias and qualitative research.............................................................................33
Key learning points.......................................................................................................................33
Chapter 9: Systematic reviews: Urban myths and fairy tales............................................................33
9.1 Systematic reviews and urban myths.....................................................................................33
9.5 Conclusion..............................................................................................................................34
, Systematic Reviews in the Social Sciences: a practical guide – Petticrew
and Roberts (2006)
Chapter 1: Why do we need systematic reviews?
1.1 Introduction
Systematic literature reviews method of making sense of large bodies of information, and means of
contributing to the answers to questions about what does and doesn’t work
- Mapping out areas of uncertainty
- Identify where little/no relevant research has been done where new studies are needed
Important to be able to tell the difference between real and assumed knowledge systematic
reviews can help with this
Reviews are important because the results of single studies are generally given much greater
credence than they merit
Most research can only be understood in context key part of context consists of the results of
other studies that tested the same hypothesis, in similar populations
Box 1.2: On the importance of knowing about uncertainty
Lao-Tze: ‘Important not only that we know what we know, but that we know what we don’t
know’.
Confucius: ‘When you know a thing, to hold that you know it; and when you don’t know a thing,
to allow that you don’t know it – this is knowledge.’
Rumsfeld: ‘There are known knowns: the things we know we know. There are also known
unknowns: we know there are some things we don’t know. There are also unknown unknowns,
the ones we don’t know we don’t know.’
One problem with interpreting and using research is that the research behind headlines is often far
removed from real-life settings may be difficult for policymakers or public to know whether the
results are to be taken seriously
- The more sceptical research-informed policymaker may simply wait patiently, on the grounds that
another researcher will soon publish a paper saying the opposite
Studies produce conflicting findings for other reasons than methodological shortcomings or authorial
bias
findings may often be statistically significant, but not socially significant
Sometimes claims with strong face validity (headlines) can be attractive to politicians, and seized on
as evidence to support a call to action.
Literature reviews may themselves be biased, and by carefully selecting which studies to review, it’s
possible to produce 2 similar reviews that come to entirely opposite conclusions.
