Lecture 7 Evaluation Designs: Qualitative Methods

Gang He

October 20, 2026

Recap lecture 6

• Quasi-experimental design
  • Instrumental variable (IV)
  • Regression discontinuity
  • Difference in differences
  • Matching
• Case:
  • Air quality and Life Expectancy
  • Alcohol access and mortality

Today’s agenda

• Case Study
• Case: NYC Gifted and Talented (G&T) Program

Capturing the wedge

Evaluation designs

• Experimental
  • Randomized controlled trial
• Quasi-experimental
  • Instrumental variable (IV)
  • Regression discontinuity (RDD)
  • Difference in differences (DiD)
• Observational (Non-experimental)
  • Time–series analysis (Pre- and post-intervention studies)
  • Cross-sectional surveys
  • Case studies

Case Study vs. Quantitative Impact Evaluation

Dimension	Quantitative Impact Evaluation	Qualitative Case Study
Main questions	Does it work? How much?	How and why does it work?
Primary focus	Causal effects and outcomes	Processes, mechanisms, context
Typical methods	RCT, RDD, DiD, IV	Interviews, observations, documents
Data	Numerical, structured	Textual, observational, documentary
Strengths	Estimates program impacts	Explains implementation and experiences
Limitations	May not explain mechanisms	Limited statistical generalization

Key takeaway: Quantitative methods estimate program impacts; qualitative case studies help explain how and why those impacts occur.

Case study questions

Item	Assignment
Exercise	Suppose you are working for GAO and have been asked whether the informed consent requirements for experimentation with human subjects are being properly implemented. Suppose further that we visit three sites where humans are used as subjects for research— a hospital, a university, and a clinic— and that we review the informed consent procedures at each site.
Question 1	Is this an application of the case study method? Why?
Question 2	If not, would case studies be appropriate for answering the question we were asked? Why?
Question 3	What is your definition of “case study”?

What is a case study?

A case study is a method for learning about a complex instance, based on a comprehensive understanding of that instance obtained by extensive description and analysis of that instance taken as a whole and in its context.

Case study

• The “Why” and “How” questions
• Exam program implementation
• Program must adapt to organizational context and local conditions

Categories

• Illustrative: descriptive, add realism and in-depth examples
• Exploratory: descriptive, defining the questions and hypotheses for later investigation
• Critical instance: critical test of an assertion about program, problem or strategy
• Program implementation: operation and norms about implementation
• Program effects: cause-and-effect relationships, involve multiple sites and multiple methods
• Cumulative: brings together findings from many case studies

Steps in case studies

• Designing the Case Study.
  • Define the research questions.
    
  • Develop a framework to identify study dimensions.
    
  • Determine the unit of analysis.
    
  • Establish a single-case or multiple-case design.
• Conducting the Case Study.
  • Develop protocols.
    
  • Select qualified staff and provide training.
    
  • Implement strategies for collecting data, typically from multiple sources.
• Analyzing the Data.
  • Create a database.
  • Use specific techniques to analyze the data.
• Preparing the Report.
  • Determine the audience and the organizational structure for the report.
    
  • Include context and data collection strategies in addition to findings.
    
  • Outline limitations or cautions.
    

Common Data Collection Methods

• Interviews: Understand individual experiences and perspectives

• Focus groups: Explore shared experiences and differences

• Observations: Examine how programs operate in practice

• Document analysis: Review policies, reports, and administrative records

• Surveys: Collect structured information from participants

Triangulation: Strengthening Evidence

Compare findings across multiple sources and methods.

Example: Why is participation in a workforce program declining?

• Interviews → Participants report transportation barriers

• Administrative data → Attendance is lower in distant neighborhoods

• Observations → Limited transportation options near program sites

Converging evidence strengthens the explanation.

Example: Case study of a food assistance program

Case Study Exercise: NYC Gifted and Talented Program

Scenario: NYC wants to evaluate the implementation, accessibility, and effectiveness of its Gifted and Talented (G&T) Program.

Group Discussion (10 minutes)

Topics	Evaluation Focus	Research Question
1	Access and Equity	What factors influence whether families apply?
2	Implementation	How do schools differ in implementing G&T?
3	Stakeholder Perspectives	How do parents and teachers define success?
4	Program Improvement	What changes could improve accessibility?

Designing case study

• Define the research questions
• Develop a framework to identify study dimensions
• Determine the unit of analysis
• Establish a single-case or multiple-case design
• Select cases or sites

Avoiding common pitfalls

Problem	Solution
Limited generalization	Sites selection Acknowledge limitation
Too much data	Research framework Protocols
Impartiality	Review interview Multiple data collection Supervision and training
Premature conclusions	Multiple data analysis Rival hypotheses
Narrative bias	Understand audience Explicit interpretation

Story

Elinor Ostrom and Case Studies

Summary

• Objectivities

• Generalization and replication

References

Newcomer, Kathryn E., Harry P. Hatry, and Joseph S. Wholey. 2015. Handbook of Practical Program Evaluation. 4th edition. Jossey-Bass.