Energy Systems Analysis - Lecture 14 Limitations of Models

Sample analytic questions

How do I know if I should trust a model and its results?
What are the methods to address uncertainties?
What are the emerging topics to improve IAMs?
How to interpret modeling results to communicate effectively?
Where to use model? How to build your model? What to say about your model?

Projections went off

IEA underestimates solar and wind

Science models are in better positions

Conditions for model robustness

Models can be accurate when they describe systems that

are observable and measurable
exhibit constancy of structure over time
exhibit constancy across variations in conditions not specified in the model
permit the collection of ample data

Social economic structural changes

Core data and assumptions, which drive results, are based on historical experiences, which can be off if structural conditions change
The exact timing and character of pivotal events and technology changes is unpredictable

Changing landsacpe of new power generation capacities

Shale gas revolution

Renewable costs

Mission impossible

Predict the unpredictable
Price the priceless

Better modeling uncertainties

Define an endpoint of analysis
List all uncertain parameters
Specify maximum range of values
Specify subjective probability distribution for values within the range
Determin and account for correlations
Probability distribution of model predictions
Derive quantitative uncertainties
Obtain additional data and repeat analysis if needed

Incoporating political economy

Examples

Risk averse investors
Subsidies or taxes
Carbon lock-in
Unequal costs and benefits of climate polices accrue to different groups
Public opinion
Confidence in political institutions
Trade and investments
Competence of goverment

Why a model in the first place

Description
Explanation
Experimentation
Providing sources of analogy
Communication/Education
Providing focal objects or centerpiece for scientific dialogue
Thought experiment
Projection

Modeling for insights

Modeling based on research questions
Modeling elements, structure, relations
Do not over-interpret
Do not misinterpret
“All models are wrong, but some are useful”. Avoid useless models.

Focusing on structures and relations

Knowing limitations

Known unknowns
Unknown unknowns
“Garbage in, garbage out”

Interpreting results

Results are outcomes of assumptions and models
Results does not automatically translate to policies
Communicating the limitations and uncertainties

Open source

Open model (code)
Open data
Open results
Open validation

In Open we Trust!

Ensure model for society

Mind the assumptions
Mind the hubris
Mind the framing
Mind the consequences
Mind the unknowns
Questions not answers

Summary

Know the limitation of models
Use model appropriately
Interpret/communicate the results effectively
Models for social good

References

Craig, Paul P, Ashok Gadgil, and Jonathan G Koomey. 2002. “What Can History Teach Us? A Retrospective Examination of Long-Term Energy Forecasts for the United States.” Annual Review of Energy and the Environment 27 (1): 83–118. https://doi.org/10.1146/annurev.energy.27.122001.083425.

Hausfather, Zeke, Henri F Drake, Tristan Abbott, and Gavin A Schmidt. 2020. “Evaluating the Performance of Past Climate Model Projections.” Geophysical Research Letters 47 (1): e2019GL085378. https://doi.org/10.1029/2019GL085378.

Hodges, James S, James A Dewar, et al. 1992. “Is It You or Your Model Talking?: A Framework for Model Validation.” Rand Santa Monica, CA. https://www.rand.org/pubs/reports/R4114.html.

Peng, Wei, Gokul Iyer, Matthew Binsted, Jennifer Marlon, Leon Clarke, James A Edmonds, and David G Victor. 2021. “The Surprisingly Inexpensive Cost of State-Driven Emission Control Strategies.” Nature Climate Change 11 (9): 738–45. https://doi.org/10.1038/s41558-021-01128-0.

Peng, Wei, Gokul Iyer, Valentina Bosetti, Vaibhav Chaturvedi, James Edmonds, Allen A Fawcett, Stéphane Hallegatte, David G Victor, Detlef van Vuuren, and John Weyant. 2021. “Climate Policy Models Need to Get Real about People—Here’s How.” Nature Publishing Group. https://doi.org/10.1038/d41586-021-01500-2.

Saltelli, Andrea, Gabriele Bammer, Isabelle Bruno, Erica Charters, Monica Di Fiore, Emmanuel Didier, Wendy Nelson Espeland, et al. 2020. “Five Ways to Ensure That Models Serve Society: A Manifesto.” Nature Publishing Group. https://doi.org/10.1038/d41586-020-01812-9.