Lecture 6 Evaluation Designs: Quasi Experiments
Gang He
March 11, 2025
Recap lecture 5
- Evaluation design
- RCT
- Case: Microfinance and RCT
- Proposal
Today’s agenda
- Quasi-experimental design
- Regression discontinuity
- Difference in differences
- Matching
- Case: Air quality and Life Expectancy
Capturing the wedege
Source: Andrew Wheiss
Evaluation designs
- Experimental
- Randomised controled trial
- Quasi-experimental
- Instrumental variable (IV)
- Regression discontinuity (RDD)
- Difference in differences (DiD)
- Observational (Non-experimental)
- Time–series analysis (Pre- and post-intervention studies)
- Cross-sectional surveys
- Case studies
Instrumental variable design
An instrumental variable:
- It influences the likelihood of participating in a program
- Outside of the participant’s control
- Unrelated to the participant’s characteristics
Examples:
- Test scores
- Geographic proximity
- Month of birth
- Rainfall
RDD design
Application: programs that have a continuous eligibility index with clearly defined eligibility threshold to determine who is eligible and who is not.
- The index must rank people or units in continuous way.
- The index must have a clearly defined cutoff score.
- The cutoff must be unique to the program of interest.
- The score of particular individual or unit cannot be manipulated.
RDD example: Farm baseline
Source: Gertler et al. (2016)
RDD example: Farm followup
Source: Gertler et al. (2016)
Heating, air quality, and life-expectancy
Source: Chen et al. (2013)
DiD design
- Compares the CHANGES in outcomes over time between the treatment group the comparison group.
- First Difference: time-varying factors
- Second Difference: treatment effects
- Trend differences
DiD example: Employment
Source: Gertler et al. (2016)
Equal trends in DiD
Source: Gertler et al. (2016)
Testing for equal trends in DiD
- Repeated comparisons: year -2, -1, and 0
- Placebo test: a fake treatment group
- Fake treatment group + Fake outcome
- Different comparison group
Matching
Matching uses large data sets and statistical techniques to construct the best possible comparison group based on observed characteristics.
For every possible unit under treatment, it attempts to fi nd a nontreatment unit (or set of nontreatment units) that has the most similar characteristics possible.
Curse of dimensionality
Increased matching dimensions reduces matching possibilities.
Source: Gertler et al. (2016)
Propensity score matching
Uses a probability score to find the best possible comparison group.
Source: Gertler et al. (2016)
Key issues in matching
- Matching can use only observed characteristics
- Using only characteristics that are not affected by the program
- Eesimation are only as good as the characteristics used for matching (understanding the criteria for participation)
References
Chen, Yuyu, Avraham Ebenstein, Michael Greenstone, and Hongbin Li. 2013. “Evidence on the Impact of Sustained Exposure to Air Pollution on Life Expectancy from China’s Huai River Policy.” Proceedings of the National Academy of Sciences, July. https://doi.org/10.1073/pnas.1300018110.
Gertler, Paul J., Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch. 2016. Impact Evaluation in Practice, Second Edition. Washington, DC: World Bank. https://doi.org/10.1596/978-1-4648-0779-4.