Program Evaluation – Lecture 3 Logic and Causal Models

Recap lecture 2

Types and applications of evaluation
Exploratory evaluation
High Line Park
Cultural responsive evaluation
Northwest Housing Alternatives

Today’s agenda

Theory of change
Logic model
DAG: Directed Acyclic Graphs
Causal models
Jobs Plus in NYC case
Ask a good question game

Theory of change

flowchart LR
  A(IRA) --> B(Subsidies)
  B --> C(Investment)
  B --> D(Jobs)
  C <--> D
  C --> E(Domestic Manufacturing)
  D --> E(Domestic Manufacturing)

Elements of a program

Inputs:

Things that go into an activity; money, people, time, etc.

Activities:

Actions that convert inputs to outputs; things that the program does

Outputs:

Tangible goods and services produced by activities; you have control over these

Outcomes:

What happens when the target population uses the outputs; you don’t have control over these

Inputs → Activities → Outputs → Outcomes → Final outcomes

Capturing the wedege

Data

Experimental: you have control over which units receive treatment
Observational: you do not have control over which units receive treatment

Natural experiments

Real experiments could be

High costs
Infeasible
Unethical

Stories

Dell and Querubin (2018)
Chen et al. (2013)

Directed Acyclic Graphs (DAG)

Directed: Each node has an arrow that points to another node

Acyclic: You can’t cycle back to a node (and arrows only have one direction)

Graph: It’s a graph

Draw a DAG

Step 1: List variables

Step 2: Simplify

Step 3: Connect arrows

Step 4: Use logic and math to determine which nodes and arrows to measure

What is the causal effect of an additional year of education on earnings?

Causal identification

A causal effect is identified if the association between treatment and outcome is propertly stripped and isolated.

Arrows in a DAG transmit associations
We can redirect and control those paths by “adjusting” or “conditioning”

Three types of associations

Confounding: Common cause

Causation: Mediation

Collision: Endogeneity/Selection

Confounding example

What’s the relations between money and win margin?

Money \(\rightarrow\) Win
Money \(\leftarrow\) Quality \(\rightarrow\) Win
Quality is a backdoor

Solution:

Find the part of campaign money that is explained by quality, remove it. This is the residual part of money.
Find the part of win margin that is explained by quality, remove it. This is the residual part of win margin.
Find the relationship between the residual part of money and residual part of win margin. This is the causal effect.

Causasion example

Should you control job connections?

Avoid overcontrolling

Colliders example

Do programming skills reduce social skills?

Hired by a tech company inadvertently connected the two.

Colliders example

Height is unrelated to basketball skill among NBA players

Colliders can create fake causal effects
Colliders can hide real causal effects

Counterfacture, intervention, and effects

Control backdoors
Average treatment effect
Sub groups (age, race, ethnicity, income, etc.)

Causal models

Instrumental variables
Randomized controlled trail (RCT)
Regressional discontinuity (RD)
Diference in difference (DiD)
Matching

References

Chen, Yuyu, Avraham Ebenstein, Michael Greenstone, and Hongbin Li. 2013. “Evidence on the Impact of Sustained Exposure to Air Pollution on Life Expectancy from China’s Huai River Policy.” Proceedings of the National Academy of Sciences, July. https://doi.org/10.1073/pnas.1300018110.

Dell, Melissa, and Pablo Querubin. 2018. “Nation Building Through Foreign Intervention: Evidence from Discontinuities in Military Strategies.” The Quarterly Journal of Economics 133 (2): 701–64. https://doi.org/10.1093/qje/qjx037.

Gertler, Paul J., Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch. 2016. Impact Evaluation in Practice, Second Edition. Washington, DC: World Bank. https://doi.org/10.1596/978-1-4648-0779-4.