Lecture 5 Evaluation Designs: Randomized Experiments

Gang He

March 4, 2025

Recap lecture 4

• Stakeholders
• Identify key stakeholder
• Engaging stakeholders
• ARPR-E case

Today’s agenda

• Evaluation design
• RCT
• Case: Microfinance and RCT
• Proposal

Capturing the wedege

Counterfactural: What if?

What would have happened in the absence of the program?

How do we measure what would have happened if the other circumstance had prevailed?

Comparison group

A valid comparison group:

1. has the same characteristics, on average, as the treatment group in the absence of the program;
2. remains unaffected by the program; and
3. would react to the program in the same way as the treatment group, if given the program.

Note: Individual level effect is impossible to observe. No individual counterfactuals.

Evaluation designs

• Experimental
  • Randomised controled trial
• Quasi-experimental
  • Regression discontinuity
  • Difference in differences
• Observational (Non-experimental)
  • Time–series analysis (Pre- and post-intervention studies)
  • Cross-sectional surveys
  • Case studies

Naive design

• Naturally occurring treated and untreated groups
• Examples
  • Students who participated pre-K (test scores)
  • Moms who use anesthetic (hospital stays)
• Questions to ask: Do we expect that if the program had not been offered, the outcomes of the treated and untreated comparison groups would have been roughly the same?

Challenges: Counterfeits

• Self-selection
• Selection bias

The power of randomized

Comparing average outcomes only works if groups that received/didn’t receive treatment look the same.

Why randomize

• Internal and external validity
• Fairness
• Acceptance
• “Gloden Standard”

Random assignment methods

• Coins
• Dice
• Lottery
• Random numbers

Tool: Random Generator

Steps in RCTs

What can be randomized?

Different ways to randomize come from three basic elements of a program which can be randomized

• Access: which people are offered access to a program

• Timing: when people are offered access

• Encouragement: which people are given encouragement to participate

When to use RCTs?

• Demand for treatment exceeds supply
• Treatment will be phased in over time
• Treatment has uncertainty
• Local culture open to randomization
• When people won’t know and it’s ethical
• When lotteries are going to happen anyway

When not?

• When you need immediate results
• When it’s unethical or illegal
• When it’s something that happened in the past
• When it involves universal ongoing phenomena

Abdul Latif Jameel Poverty Action Lab

Story

Esther Duflo and RCTs

Rule of thumb

• Correlation is the beginning of understanding what happens to a group
• Correlation does not equal causation
• Evaluation design will depends on possibility, feasibility, and appropriability

References

Gertler, Paul J., Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch. 2016. Impact Evaluation in Practice, Second Edition. Washington, DC: World Bank. https://doi.org/10.1596/978-1-4648-0779-4.