How andrewlyden/pypsa-gb works
PyPSA-GB is a Snakemake-orchestrated power-systems model centered on a PyPSA network object. The repo is organized around turning raw GB energy-system datasets into a solved network for either historical analysis or future scenarios, then validating and analyzing the results.
1. Top-level orchestration
The main entry point is Snakefile, with domain-specific rule sets in rules/. These rules compose a pipeline from source data ingestion through network assembly, scenario augmentation, solving, and analysis. Snakefile_cutouts appears to handle a separate weather/cutout preparation workflow used by renewables and flexibility modules.
2. Configuration and scenario routing
config/ is the control plane. config.yaml, scenarios.yaml, defaults.yaml, and the loader in config_loader.py define which scenario runs, which model variant is active, and how inputs are parameterized. The repo explicitly routes years <= 2024 through historical data paths and years > 2024 through future projections, which is a key architectural boundary.
3. Data lake / source-data layer
data/ contains the curated inputs the pipeline depends on. The major source families are:
- FES: future scenario assumptions and API-derived tables
- DUKES / generators: plant metadata, fuel, emissions, marginal costs, geocoding caches
- Demand: ESPENI and related demand profiles
- Interconnectors: NESO registers, DUKES interconnector data, bus mappings
- Market: Elexon bid/offer data and processed calibration artifacts
- Network: ETYS, reduced network, and zonal network definitions
- Renewables / storage / LOLE / validation: technology-specific and check datasets
4. Core model boundary: network construction
The central product of the pipeline is a PyPSA network. Build logic lives in scripts/network_build/ and rules/network_build.smk, which assemble base topology and ETYS-derived infrastructure. The repo supports three network abstractions, each with a different granularity:
- ETYS full network: detailed, 2000+ buses
- Reduced network: compact representation for faster runs
- Zonal network: coarse 17-zone model for higher-level studies
5. Transmission and geography mapping
A large part of the architecture is spatial mapping between physical assets and network nodes. scripts/network_build/, scripts/utilities/spatial_utils.py, and data in data/network/ tie together substations, GSP regions, ETYS boundaries, bus mappings, and shapefiles/GeoJSON. This layer is what makes location-sensitive modeling possible for generators, interconnectors, demand, and upgrades.
6. Generation subsystem
scripts/generators/ and rules/generators.smk transform raw generator registries into model-ready assets. It covers thermal, renewable, and dispatchable generators, crosswalks between source registries, capacity aggregation, location enrichment, and marginal cost assignment. This subsystem also integrates historical and future retirement/new-build assumptions.
7. Demand, flexibility, and storage subsystems
Demand is handled in scripts/demand/ with profiles and flexibility additions for heat pumps, EVs, and other demand-side response mechanisms. Storage is handled in scripts/storage/, including battery and external storage parameterization. These modules are separate because they inject time-series behavior and flexible demand/dispatch constraints into the network after the base topology is created.
8. Renewables and weather-dependent inputs
scripts/renewables/ builds renewable profiles and maps them onto sites or buses, including offshore, marine, hydro, and geothermal-related workflows. The presence of Snakefile_cutouts, download_cutouts.py, and cutout-related utilities suggests weather cutouts are an external dependency for time-varying renewable availability and related time-series preparation.
9. Interconnectors, hydrogen, and market modules
Interconnectors have their own ingestion/mapping pipeline in scripts/interconnectors/, reflecting their special role as cross-border assets with register data, flow validation, and price/availability enrichment. Hydrogen is represented as an add-on system in scripts/hydrogen/. Market modeling is split into scripts/market/, with Elexon calibration, balancing-market/wholesale analysis, and validation logic; data/market/ stores both raw BOD records and processed ladders/bids/offers artifacts.
10. Solve and post-process
scripts/solve/ contains the optimization boundary: building the final solvable network, applying hydro constraints, solving, and finalizing outputs. This is the point where all prior integrations converge into an optimization problem. The outputs then feed analysis and validation scripts, rather than back into construction.
11. Validation and analysis boundary
scripts/analysis/ and tests/workflow/ encode the model-checking and comparison layer. This includes validation against historical market and constraint data, network integrity checks, scenario comparison, and notebook generation for deep dives. The tests are organized to cover both unit-level transformations and end-to-end historical/future scenario pipelines.
12. Documentation and reproducibility
docs/ is a full Sphinx/nbsphinx documentation site with tutorials that mirror the main workflows: demand, generators, networks, renewables, storage, interconnectors, hydrogen, and market dispatch. envs/pypsa-gb.yaml defines the runtime environment, making the repo reproducible across the Snakemake pipeline, notebooks, and tests.
13. Primary data flow to diagram
A correct architecture diagram should show: external source datasets and caches → domain preprocessing packages (generators, demand, renewables, storage, interconnectors, network_build, market) → assembled PyPSA network → solve/optimization → analysis/validation outputs. The key conceptual boundary is that source-specific ETL lives in scripts/, while the model itself is the network constructed and solved by Snakemake rules.
Main entry points: build_network.py Build network · solve_network.py Solve · finalize_network.py Finalize · config_loader.py Config
Components 26
Orchestration
Snakemake entrypoints and rule sets that drive the model pipeline.
Configuration
Scenario selection, defaults, and runtime configuration.
- Config · Loads scenario, defaults, and year-based model routing.
config/config_loader.py - YAML config · Scenario and defaults definitions controlling model variants.
config/config.yaml
Data
Curated input datasets used by the pipeline.
- FES · Future energy scenario assumption tables and API-derived inputs.
data/FES - Generators · Plant registries, fuel, emissions, coordinates, and crosswalks.
data/generators - Demand · Demand profiles and baseline demand tables.
data/demand - Network data · ETYS, reduced-network, zonal, and boundary mapping inputs.
data/network - Renewables · Renewable planning and marine/offshore site datasets.
data/renewables - Market data · Elexon bid/offer records and calibration artifacts.
data/market - Interconnectors · Registers, mappings, and flow-related inputs for cross-border assets.
data/interconnectors
Preprocess
Domain ETL modules that transform raw datasets into model-ready assets.
- Demand ETL · Builds demand profiles and flexibility additions.
scripts/demand/integrate.py - Gen ETL · Maps generators, enriches locations, and assigns capacities/costs.
scripts/generators/finalize_generator_integration.py - Renewables ETL · Generates and maps renewable availability profiles to the network.
scripts/renewables/renewable_integration.py - Storage ETL · Adds storage assets and parameters into the model.
scripts/storage/add_storage.py - Interconnector ETL · Cleans, maps, and adds interconnectors to the network.
scripts/interconnectors/add_to_network.py - Market ETL · Processes Elexon market data and validates balancing-market logic.
scripts/market/elexon_data.py - Hydrogen · Adds hydrogen system representation to the network.
scripts/hydrogen/add_hydrogen_system.py
Network
Core network construction and clustering boundary.
- Build network · Assembles the PyPSA network from ETYS and spatial mappings.
scripts/network_build/build_network.py - Cluster · Aggregates components into reduced or zonal network forms.
scripts/network_clustering/cluster_network.py
Solve
Optimization and final network preparation.
Analysis
Validation, comparison, and result analysis.
Docs
Documentation and tutorial workflows.
- Docs · Sphinx/nbsphinx documentation and tutorials mirroring workflows.
docs/source/index.rst
Connections 26
- Snakefile → Rules: orchestrates · Main workflow delegates execution to rule sets.
- Cutouts → Rules: supports · Cutout preparation feeds renewable and flexibility workflows.
- YAML config → Config: loads · Configuration files are interpreted by the loader.
- Config → Snakefile: routes · Scenario and year selection drive the pipeline path.
- FES → Config: scenario inputs · Future scenario assumptions parameterize model runs.
- Generators → Gen ETL: inputs · Raw generator registries and caches feed generator ETL.
- Demand → Demand ETL: inputs · Demand profiles are transformed into model demand series.
- Network data → Build network: inputs · Network topology and spatial boundaries feed assembly.
- Renewables → Renewables ETL: inputs · Renewable site and planning datasets feed profile generation.
- Market data → Market ETL: inputs · Elexon market records feed calibration and validation.
- Interconnectors → Interconnector ETL: inputs · Register and mapping data feed interconnector ETL.
- Demand ETL → Build network: injects · Demand loads are attached to the assembled network.
- Gen ETL → Build network: injects · Generators are mapped into the network model.
- Renewables ETL → Build network: injects · Renewable assets and profiles are attached to buses.
- Storage ETL → Build network: injects · Storage assets become part of the network.
- Interconnector ETL → Build network: injects · Cross-border assets are connected to the network.
- Hydrogen → Build network: injects · Hydrogen system components extend the network.
- Market ETL → Solve: calibrates · Market-derived parameters influence solving and constraints.
- Build network → Cluster: reduces · Built network can be clustered into reduced abstractions.
- Cluster → Solve: prepares · Clustered networks are handed to the solver.
- Build network → Solve: solves · The assembled network is the core optimization input.
- Solve → Finalize: outputs · Solved results are finalized for downstream use.
- Finalize → Analyze: analyzed by · Final outputs feed result analysis and comparison.
- Finalize → Validate: validated by · Solved networks are checked against historical and scenario tests.
- Analyze → Docs: documented in · Analysis workflows are mirrored in tutorials and docs.
- Validate → Docs: documented in · Validation workflows and tutorials are described in docs.