Lecture 14 Limitations of Models

Gang He

November 28, 2022

Sample analytic questions

  • How do I know if I should trust a model and its results?
  • What are the methods to address uncertainties?
  • What are the emerging topics to improve IAMs?
  • How to interpret modeling results to communicate effectively?
  • Where to use model? How to build your model? What to say about your model?

Projections went off

IEA underestimates solar and wind

Science models are in better positions

Conditions for model robustness

Models can be accurate when they describe systems that

  • are observable and measurable
  • exhibit constancy of structure over time
  • exhibit constancy across variations in conditions not specified in the model
  • permit the collection of ample data

Social economic structural changes

  • Core data and assumptions, which drive results, are based on historical experiences, which can be off if structural conditions change
  • The exact timing and character of pivotal events and technology changes is unpredictable

Changing landsacpe of new power generation capacities

Shale gas revolution

Renewable costs

Mission impossible

  • Predict the unpredictable
  • Price the priceless

Better modeling uncertainties

  • Define an endpoint of analysis
  • List all uncertain parameters
  • Specify maximum range of values
  • Specify subjective probability distribution for values within the range
  • Determin and account for correlations
  • Probability distribution of model predictions
  • Derive quantitative uncertainties
  • Obtain additional data and repeat analysis if needed

Incoporating political economy


  • Risk averse investors
  • Subsidies or taxes
  • Carbon lock-in
  • Unequal costs and benefits of climate polices accrue to different groups
  • Public opinion
  • Confidence in political institutions
  • Trade and investments
  • Competence of goverment

Why a model in the first place

  • Description
  • Explanation
  • Experimentation
  • Providing sources of analogy
  • Communication/Education
  • Providing focal objects or centerpiece for scientific dialogue
  • Thought experiment
  • Projection

Modeling for insights

  • Modeling based on research questions
  • Modeling elements, structure, relations
  • Do not over-interpret
  • Do not misinterpret
  • “All models are wrong, but some are useful”. Avoid useless models.

Focusing on structures and relations

Knowing limitations

  • Known unknowns
  • Unknown unknowns
  • “Garbage in, garbage out”

Interpreting results

  • Results are outcomes of assumptions and models
  • Results does not automatically translate to policies
  • Communicating the limitations and uncertainties

Open source

  • Open model (code)
  • Open data
  • Open results
  • Open validation

In Open we Trust!

Ensure model for society

  • Mind the assumptions
  • Mind the hubris
  • Mind the framing
  • Mind the consequences
  • Mind the unknowns
  • Questions not answers


  • Know the limitation of models
  • Use model appropriately
  • Interpret/communicate the results effectively
  • Models for social good


Craig, Paul P, Ashok Gadgil, and Jonathan G Koomey. 2002. “What Can History Teach Us? A Retrospective Examination of Long-Term Energy Forecasts for the United States.” Annual Review of Energy and the Environment 27 (1): 83–118. https://doi.org/10.1146/annurev.energy.27.122001.083425.
Hausfather, Zeke, Henri F Drake, Tristan Abbott, and Gavin A Schmidt. 2020. “Evaluating the Performance of Past Climate Model Projections.” Geophysical Research Letters 47 (1): e2019GL085378. https://doi.org/10.1029/2019GL085378.
Hodges, James S, James A Dewar, et al. 1992. “Is It You or Your Model Talking?: A Framework for Model Validation.” Rand Santa Monica, CA. https://www.rand.org/pubs/reports/R4114.html.
Peng, Wei, Gokul Iyer, Matthew Binsted, Jennifer Marlon, Leon Clarke, James A Edmonds, and David G Victor. 2021. “The Surprisingly Inexpensive Cost of State-Driven Emission Control Strategies.” Nature Climate Change 11 (9): 738–45. https://doi.org/10.1038/s41558-021-01128-0.
Peng, Wei, Gokul Iyer, Valentina Bosetti, Vaibhav Chaturvedi, James Edmonds, Allen A Fawcett, Stéphane Hallegatte, David G Victor, Detlef van Vuuren, and John Weyant. 2021. “Climate Policy Models Need to Get Real about People—Here’s How.” Nature Publishing Group. https://doi.org/10.1038/d41586-021-01500-2.
Saltelli, Andrea, Gabriele Bammer, Isabelle Bruno, Erica Charters, Monica Di Fiore, Emmanuel Didier, Wendy Nelson Espeland, et al. 2020. “Five Ways to Ensure That Models Serve Society: A Manifesto.” Nature Publishing Group. https://doi.org/10.1038/d41586-020-01812-9.