Cross-validation
Give an independent program our exact inputs. Does it get our answers?
EVIDENCE · FIRST-CLASS
Validation checks the model against what the grid actually did. This page checks it against other people’s code. PyPSA is the standard open-source power-system model, maintained at TU Berlin and used by research groups worldwide. We did not write it, and it shares no code with our engine. We exported our scenario data bit-for-bit, fed it to PyPSA, and asked it the same questions. Same inputs, independent implementation: if the two programs disagree, one of them is wrong.
Both programs dispatch the same fleet the same way
The 2024 reference year. Both programs decide, half-hour by half-hour, which plant runs:
Annual gas, our engine 73,026.751GWh
Annual gas, PyPSA 73,026.751GWh
Difference 0.000MWh
Half-hours identical 97.3%
17,087 of 17,568 half-hours agree to better than 1 kW. Full comparison: docs/notes/pypsa-xval-findings.md in the engine repository.
The remaining 481 half-hours are cases where two dispatch schedules cost exactly the same and each solver picked one at random. They net to zero energy.
The comparison did turn up seven differences, and we published all of them. Each traces to a design choice already documented on this site, not to a bug in either program. The largest: our standard dispatch rule has no foresight, because a real grid operator cannot see next week’s weather. An optimiser like PyPSA plans the whole year with hindsight, so it pre-charges storage before scarcity and saves about 0.4 TWh of gas, 0.6% of the year’s burn. That gap is the price of honesty about foresight, and we report it rather than tune it away.
Both programs size the backup fleet the same way
The model’s capacity numbers come from a simple search: pick a backup fleet size, run the whole period, check for blackouts, adjust, repeat. We made PyPSA run the identical search on the two hardest spells in the 40-year weather record. 2010 is the worst single year. 1985 to 1987 is the worst three-year stretch.
| Spell | Our engine | PyPSA |
|---|---|---|
| 2010 | 141.8 GW | 141.8 GW |
| 1985–87 | 146.6 GW | 146.6 GW |
The agreement is not just at the end. At every step of the search, both programs computed the same blackout shortfall for the same candidate fleet, matching to about twelve significant figures. This checks the question’s formulation. The 40-year headline figures themselves rest on the 2024 outturn validation and the engine’s own paired dispatch policies.
What our simplifications cost, measured
Our engine lets a power station run at any output and switch off at will. Real stations cannot: a nuclear plant will not run below about half power, and thermal plant must stay on, or off, for hours at a time. PyPSA can model those constraints. Our engine deliberately does not, and instead of leaving that as a caveat, we used PyPSA to measure it: the two hardest years, run with the constraints off and then on.
On blackouts the constraints barely register. Unserved energy in 2010 and 2022 moves by roughly 24 GWh against shortfalls of tens of thousands: 0.04% to 0.09%. In the scarcity hours that set capacity requirements, every station is already running flat out, so the freedom to switch off is worth nothing.
Curtailment is different. Stations that cannot switch off keep generating through windy nights, and their output displaces wind and solar. The constrained runs spill 25 to 34 TWh more renewable energy per hard year. Our curtailment figures for hard years are therefore lower bounds, and we quote them as such.
These two runs use published estimates of plant flexibility (minimum outputs, minimum run times) that we have documented but not yet finalised. The figures are for 2010 and 2022 alone, not for the full 40-year record. Details: docs/notes/pypsa-xval-uc-params.md in the engine repository.
What PyPSA does that we don’t, and the reverse
PyPSA can design an investment plan: tell it the costs and it picks the cheapest fleet to build. We do not use that. This project evaluates the fleets that NESO and the Climate Change Committee have already published, on their own build schedules, so a critic has to argue with NESO’s assumptions, not ours. PyPSA can also compress a year into a few representative weeks to save computation. We run every half-hour of 40 years instead, because multi-week wind droughts are the events this instrument exists to study and compression erases them. The full capability-by-capability accounting is docs/09-capability-register.md in the engine repository.
One thing does not transfer in the other direction. PyPSA has no frequency physics: no inertia, no rate-of-change-of-frequency, no loss-of-infeed events. Our stability engine models those, and there is no open framework to check it against.
- Reference implementation
- PyPSA 1.2.4 · highspy 1.15.1 · HiGHS, single-threaded
- Export
-
grid-cli export pypsa— deterministic, digest-pinned, floats exact to the bit - Harness
-
scripts/pypsa-xval/— version-locked; reads the export and nothing else - Findings
-
docs/notes/pypsa-xval-findings.md— all seven differences, each traced to its mechanism - Reports
-
pypsa-xval-tier2.md·pypsa-xval-tier3.md·pypsa-xval-spans.md