Lecture 7 Evaluation Designs: Qualitative Methods
Gang He
March 18, 2025
Recap lecture 6
- Quasi-experimental design
- Regression discontinuity
- Difference in differences
- Matching
- Case: Air quality and Life Expectancy
Today’s agenda
- Case Study
- Case: NYC Gifted and Talented (G&T) Program
Capturing the wedege
Source: Andrew Wheiss
Evaluation designs
- Experimental
- Randomised controled trial
- Quasi-experimental
- Instrumental variable (IV)
- Regression discontinuity (RDD)
- Difference in differences (DiD)
- Observational (Non-experimental)
- Time–series analysis (Pre- and post-intervention studies)
- Cross-sectional surveys
- Case studies
Case study questions
| Item | Assignment |
|---|---|
| Exercise | Suppose you are working for GAO and have been asked whether the informed consent requirements for experimentation with human subjects are being properly implemented. Suppose further that we visit three sites where humans are used as subjects for research— a hospital, a university, and a clinic— and that we review the informed consent procedures at each site. |
| Question 1 | Is this an application of the case study method? Why? |
| Question 2 | If not, would case studies be appropriate for answering the question we were asked? Why? |
| Question 3 | What is your definition of “case study”? |
Source: Adapted from GAO, Case Study Evaluations
What is a case study?
A case study is a method for learning about a complex instance, based on a comprehensive understanding of that instance obtained by extensive description and analysis of that instance taken as a whole and in its context.
Source: GAO, Case Study Evaluations
Case study
- The “Why” and “How” questions
- Exam program implementation
- Program must adapt to organizational context and local conditions
Source: Newcomer, Hatry, and Wholey (2015)
Categories
- Illustrative: descritpive, add realism and in-depth examples
- Exploratory: descritpive, defining the questions and hypotheses for later investigation
- Critical instance: critical test of an assertion about program, probelem or strategy
- Program implementation: operation and norms about implementation
- Program effects: cause-and-effect relationships, involve multiple sites and multiple methods
- Cumulative: brings together findings from many case studies
Source: Newcomer, Hatry, and Wholey (2015), GAO, Case Study Evaluations
Steps in case studies
- Designing the Case Study.
- Define the research questions.
- Develop a framework to identify study dimensions.
- Determine the unit of analysis.
- Establish a single-case or multiple-case design.
- Define the research questions.
- Conducting the Case Study.
- Develop protocols.
- Select qualified staff and provide training.
- Implement strategies for collecting data, typically from multiple sources.
- Develop protocols.
- Analyzing the Data.
- Create a database.
- Use specific techniques to analyze the data.
- Preparing the Report.
- Determine the audience and the organizational structure for the report.
- Include context and data collection strategies in addition to findings.
- Outline limitations or cautions.
- Determine the audience and the organizational structure for the report.
Source: Newcomer, Hatry, and Wholey (2015)
Example: Case study of a food assistance program
Source: Newcomer, Hatry, and Wholey (2015)
Designing case study
- Define the research questions
- Develop a framework to identify study dimensions
- Determine the unit of analysis
- Establish a single-case or multiple-case design
- Select cases or sites
Avoiding common pitfalls
| Problem | Solution |
|---|---|
| Limited generalization | Sites selection Acknowledge limitation |
| Too much data | Research framework Protocols |
| Impartiality | Review interview Multiple data collection Supervision and training |
| Premature conclusions | Multiple data analysis Rival hypotheses |
| Narrative bias | Understand audience Explicit interpretation |
Summary
Objectivities
Generalization and replication
References
Newcomer, Kathryn E., Harry P. Hatry, and Joseph S. Wholey. 2015. Handbook of Practical Program Evaluation. 4th edition. San Francisco: Jossey-Bass.
Lecture 7 Evaluation Designs: Qualitative Methods Gang He March 18, 2025