Lecture 8 Overcome Resistance and Improve Organization Capacity

Gang He

March 25, 2025

Recap lecture 7

• Case Study Evaluation
• Story: Elinor Ostrom and Case Studies
• Case: NYC Gifted and Talented (G&T) Program

Today’s agenda

• Guest Speaker: Beatrice Teston
• Overcome Resistance and Improve Organization Capacity
• Story: James March and Organization Theory
• Case: American Red Cross (ARC) Haiti Relief Program

Who cases about evaluation

• Funders
• Policy makers
• Organization

Managing an evaluation

A partnership between the research team and policy team

Research team

Technical quality and scientific integrarity

• PI, evaluation manager
• Sampling/interview/survey exert
• Data collection team

Policy team

• Policy makers
• Program implementers

Research-Policy team models

• Oursoucing evaluation (third party)
• Integration (Innovations for Poverty Action)
• Parnership models (World Bank)

General outline of an impact evaluation plan

1. Introduction
2. Description of the intervention
3. Objectives of the evaluation
   3.1 Hypotheses, theory of change, results chain
   3.2 Policy questions
   3.3 Key outcome indicators
   3.4 Risks
   
4. Evaluation design
5. Sampling and data
   5.1 Sampling strategy
   5.2 Power calculations
   
6. Preanalysis plan overview
7. Data collection plan
   7.1 Baseline survey
   7.2 Follow-up survey(s)

8. Products to be delivered
   8.1 Baseline report
   8.2 Impact evaluation report
   8.3 Policy brief
   8.4 Fully documented data sets, design and analysis protocols
   
9. Dissemination plan
10. Ethical protocols on protection of human subjects
    10.1 Ensuring informed consent
    10.2 Obtaining approval from the Institutional Review Board (IRB)
    
11. Timeline
12. Budget and funding
13. Composition and roles of evaluation team

Organization challenges

• Aligning incentives
• Making tradeoffs
• Asking the right questions
• Finding the right timing

Challenge 1: Aligning incentives

• Program implementers
• Donors

Stategies:

• Separate accountability from learning and impact
• Focus impact evaluation on important strategic questions
• Consider impact evaluation as an opportunity for R&D
• Reframe donor expectations: help donor make decision about supporting similar programs

Challenge 2: Making tradeoffs

• Resources for evalution or intervention
• Data collection costs
• Program staff evaluation requirements

Stategies:

• Senior management buy-in
• Outside resources to support evaluation
• Targetd evaluation support
• Training

Challenge 3: Asking the right questions

A good evaluation questioon is one that seeks to confirm or bettter understand the program’s theory of change and generate actionale information for decision makers.

• Clear thory of change
• Generate transportable knowledge

Stategies:

• Senior management buy-in
• Internal analytical capacity and other relevant resources
• Systematic review

Challenge 4: Find the right timing

The evaluation results will be most useful in making decisions about expanding or replicating a program, or changing important aspects of how the program is delivered.

• Time to make decision about program design
• Delays

Stategies:

• Project maturity
• Consistent implementation
• Timing of decision

Overcome challenges

• High-priority questions to the organization, with a strong organizational consensus around the need for impact evidence;
• A review of available evidence should demonstrate that there are knowledge gaps worth filling;
• A commitment to acting on the results of the evaluation;
• The program should be sufficiently mature with a strong M&E system in place;
• The costs of conducting an evaluation should be justified relative to its benefits and both costs as well as benefits should be clearly defined.

Capacity building

• Who: evaluators, managers, program implementers
  
• What: technical, organization, data, analysis, communication
  
• How: training, mentoring, coaching, collaboration, dialogue, peer led approach, community of practice

World Bank Framework

Summary

• Building partnership

• Capacity building

Story

James March and Organization Theory

References

Gertler, Paul J., Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch. 2016. Impact Evaluation in Practice, Second Edition. Washington, DC: World Bank. https://doi.org/10.1596/978-1-4648-0779-4.