Case 6 course 6 learning year 2 GW (BMEZ)
Name: Esmée Cox
Student number: i6160834
Date: 27 June 2019
Learning goals:
1. What is a process evaluation?
2. Why do we need process evaluation with health care innovations?
3. How to develop/perform process evaluation in health care?
4. What would be the key question when you evaluate the process of ‘’app’’?
5. What method would you use to evaluate the process of the decision “app”?
Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I. and Petticrew, M. (2008)
Developing and evaluating complex interventions: the new Medical Research Council
guidance. BMJ, 337. pp. 979-983
2000 MRC framework
- Recommendations for improvement
o Greater attention to early phase piloting and development work
o A less linear model of evaluation process
o Integration of process and outcome evaluation
o Recognition that complex interventions may work best if they are tailored to
local contexts rather than completely standardized
o Greater use of the insights provided by the theory of complex adaptive
systems
Complex interventions (= interventions that contain several interacting components)
- Number of interacting components within the experimental and control
interventions
- Number and difficulty of behaviors required by those delivering or receiving the
intervention
- Number of groups or organizational levels targeted by the intervention
- Number and variability of outcomes
- Degree of flexibility or tailoring of the intervention permitted
Developing and evaluating complex studies
- A good theoretical understanding is needed of how the intervention causes change,
so that weak links in the causal chain can be identified and strengthened
- Lack of effect may reflect implementation failure (or teething problems) rather than
genuine ineffectiveness; a thorough process evaluation is needed to identify
implementation problems
- Variability in individual level outcomes may reflect higher level processes; sample
sizes may need to be larger to take account of the extra variability and cluster
randomized designs considered
- A single primary outcome may not make best use of the data; a range of measures
will be needed and unintended consequences picked up where possible
Pagina 1 van 17
, - Ensuring strict standardization may be inappropriate; the intervention may work
better if a specified degree of adaptation to local settings is allowed for in the
protocol
2008 MRC framework
- Updates
o The inclusion of a model of the evaluation process less closely tied to the
phases of drug development
o More guidance on how to approach the development, reporting, and
implementation of complex interventions
o Greater attention to the contexts in which interventions take place
o Consideration of alternatives to randomized trials, and of highly complex or
non-health sector interventions to which biomedical methods may not be
applicable
o More evidence and examples to back up and illustrate the recommendations
- Developing a complex intervention
o Identifying existing evidence
Identify what is already known about similar interventions and the
methods that have been used to evaluate them
o Identifying and developing theory
Develop a theoretical understanding of the likely process of change by
drawing on existing evidence and theory, supplemented if necessary
by new primary research
o Modelling process and outcomes
A series of studies may be required to progressively refine the design
before embarking on a full scale evaluation
- Assessing feasibility
o Evaluations are often undermined by problems of acceptability, compliance,
delivery of the intervention, recruitment and retention, and smaller than
expected effect sizes that could have been predicted by thorough piloting
o A pilot study need not to be a scale model of the planned evaluation but
should examine the key uncertainties that have been identified during
development
- Evaluating a complex intervention
o Researchers should beware of blanket statements about what designs are
suitable for what kind of intervention and choose on the basis of specific
characteristics of the study
Pagina 2 van 17
Name: Esmée Cox
Student number: i6160834
Date: 27 June 2019
Learning goals:
1. What is a process evaluation?
2. Why do we need process evaluation with health care innovations?
3. How to develop/perform process evaluation in health care?
4. What would be the key question when you evaluate the process of ‘’app’’?
5. What method would you use to evaluate the process of the decision “app”?
Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I. and Petticrew, M. (2008)
Developing and evaluating complex interventions: the new Medical Research Council
guidance. BMJ, 337. pp. 979-983
2000 MRC framework
- Recommendations for improvement
o Greater attention to early phase piloting and development work
o A less linear model of evaluation process
o Integration of process and outcome evaluation
o Recognition that complex interventions may work best if they are tailored to
local contexts rather than completely standardized
o Greater use of the insights provided by the theory of complex adaptive
systems
Complex interventions (= interventions that contain several interacting components)
- Number of interacting components within the experimental and control
interventions
- Number and difficulty of behaviors required by those delivering or receiving the
intervention
- Number of groups or organizational levels targeted by the intervention
- Number and variability of outcomes
- Degree of flexibility or tailoring of the intervention permitted
Developing and evaluating complex studies
- A good theoretical understanding is needed of how the intervention causes change,
so that weak links in the causal chain can be identified and strengthened
- Lack of effect may reflect implementation failure (or teething problems) rather than
genuine ineffectiveness; a thorough process evaluation is needed to identify
implementation problems
- Variability in individual level outcomes may reflect higher level processes; sample
sizes may need to be larger to take account of the extra variability and cluster
randomized designs considered
- A single primary outcome may not make best use of the data; a range of measures
will be needed and unintended consequences picked up where possible
Pagina 1 van 17
, - Ensuring strict standardization may be inappropriate; the intervention may work
better if a specified degree of adaptation to local settings is allowed for in the
protocol
2008 MRC framework
- Updates
o The inclusion of a model of the evaluation process less closely tied to the
phases of drug development
o More guidance on how to approach the development, reporting, and
implementation of complex interventions
o Greater attention to the contexts in which interventions take place
o Consideration of alternatives to randomized trials, and of highly complex or
non-health sector interventions to which biomedical methods may not be
applicable
o More evidence and examples to back up and illustrate the recommendations
- Developing a complex intervention
o Identifying existing evidence
Identify what is already known about similar interventions and the
methods that have been used to evaluate them
o Identifying and developing theory
Develop a theoretical understanding of the likely process of change by
drawing on existing evidence and theory, supplemented if necessary
by new primary research
o Modelling process and outcomes
A series of studies may be required to progressively refine the design
before embarking on a full scale evaluation
- Assessing feasibility
o Evaluations are often undermined by problems of acceptability, compliance,
delivery of the intervention, recruitment and retention, and smaller than
expected effect sizes that could have been predicted by thorough piloting
o A pilot study need not to be a scale model of the planned evaluation but
should examine the key uncertainties that have been identified during
development
- Evaluating a complex intervention
o Researchers should beware of blanket statements about what designs are
suitable for what kind of intervention and choose on the basis of specific
characteristics of the study
Pagina 2 van 17
