EVALUATION
A systematic approach
,Inhoud
Chapter one: What Is Program Evaluation and Why Is It Needed?........................................................6
Program evaluation...........................................................................................................................6
Challenges any evaluation approach must deal with.........................................................................6
Systematic program evaluation.........................................................................................................7
Adapting to the Political and Organizational Context........................................................................7
Influencing Social Action to Improve Social Conditions.....................................................................8
The Purpose of the Evaluation...........................................................................................................8
Reasons for an evaluation..................................................................................................................9
Stakeholders....................................................................................................................................10
Criteria for program performance...................................................................................................11
The Five Domains of Evaluation Questions and Methods................................................................12
Need for the Program: Needs Assessment......................................................................................12
Assessment of Program Theory and Design.....................................................................................12
Assessment of Program Process......................................................................................................13
Effectiveness of the Program: Impact Evaluation............................................................................14
Cost Analysis and Efficiency Assessment.........................................................................................15
The Interplay Among the Evaluation Domains.................................................................................15
Critical Thinking/Discussion Questions............................................................................................17
Chapter 2: Social Problems and Assessing the Need for a Program....................................................18
The Role of Evaluators in Diagnosing Social Conditions and Service Needs.....................................18
Defining the Problem to Be Addressed............................................................................................19
Phases of a needs assessment.........................................................................................................19
Specifying the Extent of the Problem: When, Where, and How Big?...............................................19
Using Existing Data Sources to Develop Estimates and Identify Trends in Social Indicators............20
Estimating Problem Parameters Through Social Research..............................................................20
Forecasting needs............................................................................................................................21
Defining and Identifying the Target Populations of Interventions...................................................21
Target Boundaries............................................................................................................................22
Risk, Need, and Demand..................................................................................................................22
Incidence and Prevalence................................................................................................................22
Rates................................................................................................................................................23
Describing the Nature of Service Needs..........................................................................................23
Qualitative Methods for Describing Needs......................................................................................23
Critical Thinking/Discussion Questions............................................................................................25
1
, Application Exercises.......................................................................................................................25
Chapter 3: Assessing Program Theory and Design...............................................................................26
Evaluability Assessment...................................................................................................................26
Describing Program Theory.............................................................................................................27
Program Impact Theory...................................................................................................................28
Service Utilization Plan....................................................................................................................29
Organizational Plan..........................................................................................................................29
Eliciting Program Theory..................................................................................................................31
Defining the Boundaries of the Program.........................................................................................31
Explicating the Program Theory.......................................................................................................32
Program Goals and Objectives.........................................................................................................32
Program Functions, Components, and Activities.............................................................................32
The Logic or Sequence Linking Program Functions, Activities, and Components.............................33
Corroborating the Description of the Program Theory....................................................................33
Assessing Program Theory...............................................................................................................33
Assessment in Relation to Social Needs...........................................................................................34
Assessment of Logic and Plausibility................................................................................................34
Assessment Through Comparison With Research and Practice.......................................................36
Assessment via Preliminary Observation.........................................................................................37
Possible Outcomes of Program Theory Assessment........................................................................38
Critical Thinking/Discussion Questions............................................................................................39
Application Exercises.......................................................................................................................39
Chapter 4: Assessing Program Process and Implementation...............................................................40
What Is Program Process Evaluation and Monitoring?....................................................................40
Setting Criteria for Judging Program Process...................................................................................41
Common Forms of Process Evaluations...........................................................................................42
Process Evaluation...........................................................................................................................43
Process Monitoring and Administrative Data Systems....................................................................44
Process Assessment From the Evaluator’s Perspective...................................................................45
Process Assessment From an Accountability Perspective................................................................45
Process Assessment From a Management Perspective...................................................................45
Assessing Service Utilization............................................................................................................46
Coverage and Bias............................................................................................................................46
Measuring Coverage........................................................................................................................47
Program Records.............................................................................................................................47
Surveys............................................................................................................................................47
2
, Assessing Bias: Program Users, Eligibles, and Dropouts..................................................................48
Assessing Organizational Functions.................................................................................................48
The Delivery System........................................................................................................................49
Specification of Services..................................................................................................................49
Accessibility.....................................................................................................................................49
Program Support Functions.............................................................................................................50
Critical Thinking/Discussion Questions............................................................................................51
Application Exercises.......................................................................................................................51
Chapter 5 Measuring and Monitoring Program Outcomes..................................................................52
Program Outcomes..........................................................................................................................52
Outcome Level, Outcome Change, and Program Effect...................................................................52
Stakeholder Perspectives About Outcomes.....................................................................................54
Program Impact Theory...................................................................................................................54
Prior Research..................................................................................................................................56
Unintended Effects..........................................................................................................................56
Measuring Program Outcomes........................................................................................................56
Measurement Procedures and Properties.......................................................................................57
Reliability.........................................................................................................................................57
Validity.............................................................................................................................................58
Sensitivity.........................................................................................................................................58
Choice of Outcome Measures..........................................................................................................58
Monitoring Program Outcomes.......................................................................................................59
Indicators for Outcome Monitoring.................................................................................................59
Pitfalls in Outcome Monitoring........................................................................................................59
Interpreting Outcome Data..............................................................................................................60
Critical Thinking/Discussion Questions............................................................................................61
Application Exercises.......................................................................................................................61
Chapter 6 Impact Evaluation Isolating the Effects of Social Programs in the Real World....................62
The Nature and Importance of Impact Evaluation...........................................................................62
Additional Impact Questions...........................................................................................................62
When Is an Impact Evaluation Appropriate?...................................................................................63
What Would Have Happened Without the Program?.....................................................................63
The Logic of Impact Evaluation: The Potential Outcomes Framework.............................................64
The Fundamental Problem of Causal Inference: Unavoidable Missing Data...................................65
The Validity of Program Effect Estimates.........................................................................................66
Critical Thinking/Discussion Questions............................................................................................68
3
A systematic approach
,Inhoud
Chapter one: What Is Program Evaluation and Why Is It Needed?........................................................6
Program evaluation...........................................................................................................................6
Challenges any evaluation approach must deal with.........................................................................6
Systematic program evaluation.........................................................................................................7
Adapting to the Political and Organizational Context........................................................................7
Influencing Social Action to Improve Social Conditions.....................................................................8
The Purpose of the Evaluation...........................................................................................................8
Reasons for an evaluation..................................................................................................................9
Stakeholders....................................................................................................................................10
Criteria for program performance...................................................................................................11
The Five Domains of Evaluation Questions and Methods................................................................12
Need for the Program: Needs Assessment......................................................................................12
Assessment of Program Theory and Design.....................................................................................12
Assessment of Program Process......................................................................................................13
Effectiveness of the Program: Impact Evaluation............................................................................14
Cost Analysis and Efficiency Assessment.........................................................................................15
The Interplay Among the Evaluation Domains.................................................................................15
Critical Thinking/Discussion Questions............................................................................................17
Chapter 2: Social Problems and Assessing the Need for a Program....................................................18
The Role of Evaluators in Diagnosing Social Conditions and Service Needs.....................................18
Defining the Problem to Be Addressed............................................................................................19
Phases of a needs assessment.........................................................................................................19
Specifying the Extent of the Problem: When, Where, and How Big?...............................................19
Using Existing Data Sources to Develop Estimates and Identify Trends in Social Indicators............20
Estimating Problem Parameters Through Social Research..............................................................20
Forecasting needs............................................................................................................................21
Defining and Identifying the Target Populations of Interventions...................................................21
Target Boundaries............................................................................................................................22
Risk, Need, and Demand..................................................................................................................22
Incidence and Prevalence................................................................................................................22
Rates................................................................................................................................................23
Describing the Nature of Service Needs..........................................................................................23
Qualitative Methods for Describing Needs......................................................................................23
Critical Thinking/Discussion Questions............................................................................................25
1
, Application Exercises.......................................................................................................................25
Chapter 3: Assessing Program Theory and Design...............................................................................26
Evaluability Assessment...................................................................................................................26
Describing Program Theory.............................................................................................................27
Program Impact Theory...................................................................................................................28
Service Utilization Plan....................................................................................................................29
Organizational Plan..........................................................................................................................29
Eliciting Program Theory..................................................................................................................31
Defining the Boundaries of the Program.........................................................................................31
Explicating the Program Theory.......................................................................................................32
Program Goals and Objectives.........................................................................................................32
Program Functions, Components, and Activities.............................................................................32
The Logic or Sequence Linking Program Functions, Activities, and Components.............................33
Corroborating the Description of the Program Theory....................................................................33
Assessing Program Theory...............................................................................................................33
Assessment in Relation to Social Needs...........................................................................................34
Assessment of Logic and Plausibility................................................................................................34
Assessment Through Comparison With Research and Practice.......................................................36
Assessment via Preliminary Observation.........................................................................................37
Possible Outcomes of Program Theory Assessment........................................................................38
Critical Thinking/Discussion Questions............................................................................................39
Application Exercises.......................................................................................................................39
Chapter 4: Assessing Program Process and Implementation...............................................................40
What Is Program Process Evaluation and Monitoring?....................................................................40
Setting Criteria for Judging Program Process...................................................................................41
Common Forms of Process Evaluations...........................................................................................42
Process Evaluation...........................................................................................................................43
Process Monitoring and Administrative Data Systems....................................................................44
Process Assessment From the Evaluator’s Perspective...................................................................45
Process Assessment From an Accountability Perspective................................................................45
Process Assessment From a Management Perspective...................................................................45
Assessing Service Utilization............................................................................................................46
Coverage and Bias............................................................................................................................46
Measuring Coverage........................................................................................................................47
Program Records.............................................................................................................................47
Surveys............................................................................................................................................47
2
, Assessing Bias: Program Users, Eligibles, and Dropouts..................................................................48
Assessing Organizational Functions.................................................................................................48
The Delivery System........................................................................................................................49
Specification of Services..................................................................................................................49
Accessibility.....................................................................................................................................49
Program Support Functions.............................................................................................................50
Critical Thinking/Discussion Questions............................................................................................51
Application Exercises.......................................................................................................................51
Chapter 5 Measuring and Monitoring Program Outcomes..................................................................52
Program Outcomes..........................................................................................................................52
Outcome Level, Outcome Change, and Program Effect...................................................................52
Stakeholder Perspectives About Outcomes.....................................................................................54
Program Impact Theory...................................................................................................................54
Prior Research..................................................................................................................................56
Unintended Effects..........................................................................................................................56
Measuring Program Outcomes........................................................................................................56
Measurement Procedures and Properties.......................................................................................57
Reliability.........................................................................................................................................57
Validity.............................................................................................................................................58
Sensitivity.........................................................................................................................................58
Choice of Outcome Measures..........................................................................................................58
Monitoring Program Outcomes.......................................................................................................59
Indicators for Outcome Monitoring.................................................................................................59
Pitfalls in Outcome Monitoring........................................................................................................59
Interpreting Outcome Data..............................................................................................................60
Critical Thinking/Discussion Questions............................................................................................61
Application Exercises.......................................................................................................................61
Chapter 6 Impact Evaluation Isolating the Effects of Social Programs in the Real World....................62
The Nature and Importance of Impact Evaluation...........................................................................62
Additional Impact Questions...........................................................................................................62
When Is an Impact Evaluation Appropriate?...................................................................................63
What Would Have Happened Without the Program?.....................................................................63
The Logic of Impact Evaluation: The Potential Outcomes Framework.............................................64
The Fundamental Problem of Causal Inference: Unavoidable Missing Data...................................65
The Validity of Program Effect Estimates.........................................................................................66
Critical Thinking/Discussion Questions............................................................................................68
3
