Program Evaluation – Lecture 3 Logic and Causal Models

Recap lecture 2

Types and applications of evaluation
Exploratory evaluation
High Line Park
Cultural responsive evaluation
Case: Jobs Plus in NYC

Today’s agenda

Theory of change
Logic model
Counterfactuals and causal models
DAG: Directed Acyclic Graphs
Northwest Housing Alternatives

Theory of change

flowchart LR
  A(IRA) --> B(Subsidies)
  B --> C(Investment)
  B --> D(Jobs)
  C --> D
  C --> E(Domestic Manufacturing)
  D --> E(Domestic Manufacturing)

Elements of a program

Inputs:

Things that go into an activity; money, people, time, etc.

Activities:

Actions that convert inputs to outputs; things that the program does

Outputs:

Tangible goods and services produced by activities; you have control over these

Outcomes:

What happens when the target population uses the outputs; you don’t have control over these

Inputs → Activities → Outputs → Outcomes → Final outcomes

Logic model

flowchart LR
  A(Inputs) --> B(Activities) --> C(Outputs) --> D(Outcomes)

P is for “Produce”

C is for “Change”

Build a logic model for Jobs Plus in NYC

Capturing the wedege

Counterfactual

“The counterfactual is what would have happened—what the outcome (Y) would have been for a program participant—in the absence of the program (P).”

“Since we cannot directly observe the counterfactual, we must estimate it.”

A valid comparison group

has the same characteristics, on average, as the treatment group in the absence of the program;

remains unaffected by the program; and

would react to the program in the same way as the treatment group, if given the program.

Two counterfeit estimates of the counterfactual

Before-and-after comparisons (also known as pre-post or refl exive comparisons) compare the outcomes of the same group before and after participating in a program.

Challenge: Variables and conditions change

Enrolled-and-nonenrolled (or self-selected) comparisons compare the outcomes of a group that chooses to participate in a program with those of a group that chooses not to participate.

Challenge: Selection bias

Causal models

Instrumental variables
Randomized controlled trail (RCT)
Regressional discontinuity (RD)
Diference in difference (DiD)
Matching

Data

Experimental: you have control over which units receive treatment
Observational: you do not have control over which units receive treatment

Natural experiments

Real experiments could be

High costs
Infeasible
Unethical

Stories

Dell and Querubin (2018)

Directed Acyclic Graphs (DAG)

Directed: Each node has an arrow that points to another node

Acyclic: You can’t cycle back to a node (and arrows only have one direction)

Graph: It’s a graph

Draw a DAG

Step 1: List variables

Step 2: Simplify

Step 3: Connect arrows

Step 4: Use logic and math to determine which nodes and arrows to measure

What is the causal effect of an additional year of education on earnings?

Causal identification

A causal effect is identified if the association between treatment and outcome is propertly stripped and isolated.

Arrows in a DAG transmit associations
We can redirect and control those paths by “adjusting” or “conditioning”

Three types of associations

Confounding: Common cause

Causation: Mediation

Collision: Endogeneity/Selection

Confounding example

What’s the relations between money and win margin?

Money \(\rightarrow\) Win
Money \(\leftarrow\) Quality \(\rightarrow\) Win
Quality is a backdoor

Solution:

Find the part of campaign money that is explained by quality, remove it. This is the residual part of money.

Find the part of win margin that is explained by quality, remove it. This is the residual part of win margin.

Find the relationship between the residual part of money and residual part of win margin. This is the causal effect.

Causasion example

Should you control job connections?

Avoid overcontrolling

Causasion example

Should you control job connections?

Avoid overcontrolling

Colliders example

Do programming skills reduce social skills?

Hired by a tech company inadvertently connected the two.

Colliders example

Height is unrelated to basketball skill among NBA players

Colliders can create fake causal effects
Colliders can hide real causal effects

Counterfacture, intervention, and effects

Control backdoors
Average treatment effect
Sub groups (age, race, ethnicity, income, etc.)

References

Dell, Melissa, and Pablo Querubin. 2018. “Nation Building Through Foreign Intervention: Evidence from Discontinuities in Military Strategies.” The Quarterly Journal of Economics 133 (2): 701–64. https://doi.org/10.1093/qje/qjx037.

Gertler, Paul J., Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch. 2016. Impact Evaluation in Practice, Second Edition. Washington, DC: World Bank. https://doi.org/10.1596/978-1-4648-0779-4.