Lecture 7 Evaluation Designs: Qualitative Methods

Gang He

March 18, 2025

Recap lecture 6

• Quasi-experimental design
  • Regression discontinuity
  • Difference in differences
  • Matching
• Case: Air quality and Life Expectancy

Today’s agenda

• Case Study
• Case: NYC Gifted and Talented (G&T) Program

Capturing the wedege

Evaluation designs

• Experimental
  • Randomised controled trial
• Quasi-experimental
  • Instrumental variable (IV)
  • Regression discontinuity (RDD)
  • Difference in differences (DiD)
• Observational (Non-experimental)
  • Time–series analysis (Pre- and post-intervention studies)
  • Cross-sectional surveys
  • Case studies

Case study questions

Item	Assignment
Exercise	Suppose you are working for GAO and have been asked whether the informed consent requirements for experimentation with human subjects are being properly implemented. Suppose further that we visit three sites where humans are used as subjects for research— a hospital, a university, and a clinic— and that we review the informed consent procedures at each site.
Question 1	Is this an application of the case study method? Why?
Question 2	If not, would case studies be appropriate for answering the question we were asked? Why?
Question 3	What is your definition of “case study”?

What is a case study?

A case study is a method for learning about a complex instance, based on a comprehensive understanding of that instance obtained by extensive description and analysis of that instance taken as a whole and in its context.

Case study

• The “Why” and “How” questions
• Exam program implementation
• Program must adapt to organizational context and local conditions

Categories

• Illustrative: descritpive, add realism and in-depth examples
• Exploratory: descritpive, defining the questions and hypotheses for later investigation
• Critical instance: critical test of an assertion about program, probelem or strategy
• Program implementation: operation and norms about implementation
• Program effects: cause-and-effect relationships, involve multiple sites and multiple methods
• Cumulative: brings together findings from many case studies

Steps in case studies

• Designing the Case Study.
  • Define the research questions.
    
  • Develop a framework to identify study dimensions.
    
  • Determine the unit of analysis.
    
  • Establish a single-case or multiple-case design.
• Conducting the Case Study.
  • Develop protocols.
    
  • Select qualified staff and provide training.
    
  • Implement strategies for collecting data, typically from multiple sources.
• Analyzing the Data.
  • Create a database.
  • Use specific techniques to analyze the data.
• Preparing the Report.
  • Determine the audience and the organizational structure for the report.
    
  • Include context and data collection strategies in addition to findings.
    
  • Outline limitations or cautions.
    

Example: Case study of a food assistance program

Designing case study

• Define the research questions
• Develop a framework to identify study dimensions
• Determine the unit of analysis
• Establish a single-case or multiple-case design
• Select cases or sites

Avoiding common pitfalls

Problem	Solution
Limited generalization	Sites selection Acknowledge limitation
Too much data	Research framework Protocols
Impartiality	Review interview Multiple data collection Supervision and training
Premature conclusions	Multiple data analysis Rival hypotheses
Narrative bias	Understand audience Explicit interpretation

Story

Elinor Ostrom and Case Studies

Summary

• Objectivities

• Generalization and replication

References

Newcomer, Kathryn E., Harry P. Hatry, and Joseph S. Wholey. 2015. Handbook of Practical Program Evaluation. 4th edition. San Francisco: Jossey-Bass.