# andrewlyden/pypsa-gb architecture

> Architecture diagram of the GitHub repository andrewlyden/pypsa-gb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/andrewlyden/pypsa-gb
- Repository: https://github.com/andrewlyden/pypsa-gb

## Overview

PyPSA-GB is a Snakemake-orchestrated power-systems model centered on a PyPSA network object. The repo is organized around turning raw GB energy-system datasets into a solved network for either historical analysis or future scenarios, then validating and analyzing the results.

1. **Top-level orchestration**
   The main entry point is `Snakefile`, with domain-specific rule sets in `rules/`. These rules compose a pipeline from source data ingestion through network assembly, scenario augmentation, solving, and analysis. `Snakefile_cutouts` appears to handle a separate weather/cutout preparation workflow used by renewables and flexibility modules.

2. **Configuration and scenario routing**
   `config/` is the control plane. `config.yaml`, `scenarios.yaml`, `defaults.yaml`, and the loader in `config_loader.py` define which scenario runs, which model variant is active, and how inputs are parameterized. The repo explicitly routes years `<= 2024` through historical data paths and years `> 2024` through future projections, which is a key architectural boundary.

3. **Data lake / source-data layer**
   `data/` contains the curated inputs the pipeline depends on. The major source families are:
   - **FES**: future scenario assumptions and API-derived tables
   - **DUKES / generators**: plant metadata, fuel, emissions, marginal costs, geocoding caches
   - **Demand**: ESPENI and related demand profiles
   - **Interconnectors**: NESO registers, DUKES interconnector data, bus mappings
   - **Market**: Elexon bid/offer data and processed calibration artifacts
   - **Network**: ETYS, reduced network, and zonal network definitions
   - **Renewables / storage / LOLE / validation**: technology-specific and check datasets

4. **Core model boundary: network construction**
   The central product of the pipeline is a PyPSA network. Build logic lives in `scripts/network_build/` and `rules/network_build.smk`, which assemble base topology and ETYS-derived infrastructure. The repo supports three network abstractions, each with a different granularity:
   - **ETYS full network**: detailed, 2000+ buses
   - **Reduced network**: compact representation for faster runs
   - **Zonal network**: coarse 17-zone model for higher-level studies

5. **Transmission and geography mapping**
   A large part of the architecture is spatial mapping between physical assets and network nodes. `scripts/network_build/`, `scripts/utilities/spatial_utils.py`, and data in `data/network/` tie together substations, GSP regions, ETYS boundaries, bus mappings, and shapefiles/GeoJSON. This layer is what makes location-sensitive modeling possible for generators, interconnectors, demand, and upgrades.

6. **Generation subsystem**
   `scripts/generators/` and `rules/generators.smk` transform raw generator registries into model-ready assets. It covers thermal, renewable, and dispatchable generators, crosswalks between source registries, capacity aggregation, location enrichment, and marginal cost assignment. This subsystem also integrates historical and future retirement/new-build assumptions.

7. **Demand, flexibility, and storage subsystems**
   Demand is handled in `scripts/demand/` with profiles and flexibility additions for heat pumps, EVs, and other demand-side response mechanisms. Storage is handled in `scripts/storage/`, including battery and external storage parameterization. These modules are separate because they inject time-series behavior and flexible demand/dispatch constraints into the network after the base topology is created.

8. **Renewables and weather-dependent inputs**
   `scripts/renewables/` builds renewable profiles and maps them onto sites or buses, including offshore, marine, hydro, and geothermal-related workflows. The presence of `Snakefile_cutouts`, `download_cutouts.py`, and cutout-related utilities suggests weather cutouts are an external dependency for time-varying renewable availability and related time-series preparation.

9. **Interconnectors, hydrogen, and market modules**
   Interconnectors have their own ingestion/mapping pipeline in `scripts/interconnectors/`, reflecting their special role as cross-border assets with register data, flow validation, and price/availability enrichment. Hydrogen is represented as an add-on system in `scripts/hydrogen/`. Market modeling is split into `scripts/market/`, with Elexon calibration, balancing-market/wholesale analysis, and validation logic; `data/market/` stores both raw BOD records and processed ladders/bids/offers artifacts.

10. **Solve and post-process**
    `scripts/solve/` contains the optimization boundary: building the final solvable network, applying hydro constraints, solving, and finalizing outputs. This is the point where all prior integrations converge into an optimization problem. The outputs then feed analysis and validation scripts, rather than back into construction.

11. **Validation and analysis boundary**
    `scripts/analysis/` and `tests/workflow/` encode the model-checking and comparison layer. This includes validation against historical market and constraint data, network integrity checks, scenario comparison, and notebook generation for deep dives. The tests are organized to cover both unit-level transformations and end-to-end historical/future scenario pipelines.

12. **Documentation and reproducibility**
    `docs/` is a full Sphinx/nbsphinx documentation site with tutorials that mirror the main workflows: demand, generators, networks, renewables, storage, interconnectors, hydrogen, and market dispatch. `envs/pypsa-gb.yaml` defines the runtime environment, making the repo reproducible across the Snakemake pipeline, notebooks, and tests.

13. **Primary data flow to diagram**
    A correct architecture diagram should show: external source datasets and caches → domain preprocessing packages (`generators`, `demand`, `renewables`, `storage`, `interconnectors`, `network_build`, `market`) → assembled PyPSA network → `solve`/optimization → analysis/validation outputs. The key conceptual boundary is that source-specific ETL lives in `scripts/`, while the model itself is the network constructed and solved by Snakemake rules.

Main entry points: [`scripts/network_build/build_network.py`](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/network_build/build_network.py) (Build network), [`scripts/solve/solve_network.py`](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/solve/solve_network.py) (Solve), [`scripts/solve/finalize_network.py`](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/solve/finalize_network.py) (Finalize), [`config/config_loader.py`](https://github.com/andrewlyden/pypsa-gb/blob/master/config/config_loader.py) (Config)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_snakefile["Snakefile<br/>workflow entry"]
  node_snakefile_cutouts["Cutouts<br/>aux workflow"]
  node_rules_main["Rules"]
end

subgraph group_configuration["Configuration"]
  node_config_loader["Config<br/>[config_loader.py]"]
  node_config_files["YAML config<br/>scenario config<br/>[config.yaml]"]
end

subgraph group_data["Data"]
  node_fes_data[("FES<br/>scenario inputs")]
  node_generator_data[("Generators<br/>source data")]
  node_demand_data[("Demand<br/>source data")]
  node_network_data[("Network data<br/>spatial source")]
  node_renewables_data[("Renewables<br/>source data")]
  node_market_data[("Market data<br/>market source")]
  node_interconnector_data[("Interconnectors<br/>source data")]
end

subgraph group_preprocess["Preprocess"]
  node_demand_pipeline["Demand ETL<br/>demand integration<br/>[integrate.py]"]
  node_generator_pipeline["Gen ETL<br/>generator integration"]
  node_renewables_pipeline["Renewables ETL<br/>renewables integration"]
  node_storage_pipeline["Storage ETL<br/>storage integration<br/>[add_storage.py]"]
  node_interconnectors_pipeline["Interconnector ETL<br/>asset integration<br/>[add_to_network.py]"]
  node_market_pipeline["Market ETL<br/>market calibration<br/>[elexon_data.py]"]
  node_hydrogen_pipeline["Hydrogen<br/>system add-on"]
end

subgraph group_network["Network"]
  node_network_build["Build network<br/>network assembly<br/>[build_network.py]"]
  node_cluster_network["Cluster<br/>network reduction<br/>[cluster_network.py]"]
end

subgraph group_solve["Solve"]
  node_solve_network["Solve<br/>optimization<br/>[solve_network.py]"]
  node_finalize_network["Finalize<br/>postprocess"]
end

subgraph group_analysis["Analysis"]
  node_analysis_results["Analyze<br/>analysis<br/>[analyze_results.py]"]
  node_validation_pipeline["Validate<br/>validation"]
end

subgraph group_docs["Docs"]
  node_docs_site["Docs<br/>documentation<br/>[index.rst]"]
end

node_snakefile -->|"orchestrates"| node_rules_main
node_snakefile_cutouts -.->|"supports"| node_rules_main
node_config_files -->|"loads"| node_config_loader
node_config_loader -->|"routes"| node_snakefile
node_fes_data -.->|"scenario inputs"| node_config_loader
node_generator_data -->|"inputs"| node_generator_pipeline
node_demand_data -->|"inputs"| node_demand_pipeline
node_network_data -->|"inputs"| node_network_build
node_renewables_data -->|"inputs"| node_renewables_pipeline
node_market_data -->|"inputs"| node_market_pipeline
node_interconnector_data -->|"inputs"| node_interconnectors_pipeline
node_demand_pipeline -->|"injects"| node_network_build
node_generator_pipeline -->|"injects"| node_network_build
node_renewables_pipeline -->|"injects"| node_network_build
node_storage_pipeline -->|"injects"| node_network_build
node_interconnectors_pipeline -->|"injects"| node_network_build
node_hydrogen_pipeline -->|"injects"| node_network_build
node_market_pipeline -.->|"calibrates"| node_solve_network
node_network_build -->|"reduces"| node_cluster_network
node_cluster_network -->|"prepares"| node_solve_network
node_network_build -->|"solves"| node_solve_network
node_solve_network -->|"outputs"| node_finalize_network
node_finalize_network -->|"analyzed by"| node_analysis_results
node_finalize_network -.->|"validated by"| node_validation_pipeline
node_analysis_results -.->|"documented in"| node_docs_site
node_validation_pipeline -.->|"documented in"| node_docs_site

click node_snakefile "https://github.com/andrewlyden/pypsa-gb/tree/master/Snakefile"
click node_snakefile_cutouts "https://github.com/andrewlyden/pypsa-gb/tree/master/Snakefile_cutouts"
click node_rules_main "https://github.com/andrewlyden/pypsa-gb/tree/master/rules"
click node_config_loader "https://github.com/andrewlyden/pypsa-gb/blob/master/config/config_loader.py"
click node_config_files "https://github.com/andrewlyden/pypsa-gb/blob/master/config/config.yaml"
click node_fes_data "https://github.com/andrewlyden/pypsa-gb/tree/master/data/FES"
click node_generator_data "https://github.com/andrewlyden/pypsa-gb/tree/master/data/generators"
click node_demand_data "https://github.com/andrewlyden/pypsa-gb/tree/master/data/demand"
click node_network_data "https://github.com/andrewlyden/pypsa-gb/tree/master/data/network"
click node_renewables_data "https://github.com/andrewlyden/pypsa-gb/tree/master/data/renewables"
click node_market_data "https://github.com/andrewlyden/pypsa-gb/tree/master/data/market"
click node_interconnector_data "https://github.com/andrewlyden/pypsa-gb/tree/master/data/interconnectors"
click node_network_build "https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/network_build/build_network.py"
click node_cluster_network "https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/network_clustering/cluster_network.py"
click node_demand_pipeline "https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/demand/integrate.py"
click node_generator_pipeline "https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/generators/finalize_generator_integration.py"
click node_renewables_pipeline "https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/renewables/renewable_integration.py"
click node_storage_pipeline "https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/storage/add_storage.py"
click node_interconnectors_pipeline "https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/interconnectors/add_to_network.py"
click node_market_pipeline "https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/market/elexon_data.py"
click node_hydrogen_pipeline "https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/hydrogen/add_hydrogen_system.py"
click node_solve_network "https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/solve/solve_network.py"
click node_finalize_network "https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/solve/finalize_network.py"
click node_analysis_results "https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/analysis/analyze_results.py"
click node_validation_pipeline "https://github.com/andrewlyden/pypsa-gb/blob/master/tests/workflow/test_scenario_validation.py"
click node_docs_site "https://github.com/andrewlyden/pypsa-gb/blob/master/docs/source/index.rst"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_snakefile,node_snakefile_cutouts,node_rules_main,node_analysis_results,node_validation_pipeline toneBlue
class node_config_loader,node_config_files,node_docs_site toneAmber
class node_fes_data,node_generator_data,node_demand_data,node_network_data,node_renewables_data,node_market_data,node_interconnector_data toneMint
class node_demand_pipeline,node_generator_pipeline,node_renewables_pipeline,node_storage_pipeline,node_interconnectors_pipeline,node_market_pipeline,node_hydrogen_pipeline toneRose
class node_network_build,node_cluster_network toneIndigo
class node_solve_network,node_finalize_network toneTeal
```

## Components

### Orchestration

Snakemake entrypoints and rule sets that drive the model pipeline.

- [Snakefile](https://github.com/andrewlyden/pypsa-gb/tree/master/Snakefile): workflow entry · Main Snakemake pipeline entrypoint. · `Snakefile`
- [Cutouts](https://github.com/andrewlyden/pypsa-gb/tree/master/Snakefile_cutouts): aux workflow · Separate workflow for weather cutout preparation. · `Snakefile_cutouts`
- [Rules](https://github.com/andrewlyden/pypsa-gb/tree/master/rules): pipeline rules · Domain-specific rule sets composing the end-to-end workflow. · `rules`

### Configuration

Scenario selection, defaults, and runtime configuration.

- [Config](https://github.com/andrewlyden/pypsa-gb/blob/master/config/config_loader.py): config routing · Loads scenario, defaults, and year-based model routing. · `config/config_loader.py`
- [YAML config](https://github.com/andrewlyden/pypsa-gb/blob/master/config/config.yaml): scenario config · Scenario and defaults definitions controlling model variants. · `config/config.yaml`

### Data

Curated input datasets used by the pipeline.

- [FES](https://github.com/andrewlyden/pypsa-gb/tree/master/data/FES): scenario inputs · Future energy scenario assumption tables and API-derived inputs. · `data/FES`
- [Generators](https://github.com/andrewlyden/pypsa-gb/tree/master/data/generators): source data · Plant registries, fuel, emissions, coordinates, and crosswalks. · `data/generators`
- [Demand](https://github.com/andrewlyden/pypsa-gb/tree/master/data/demand): source data · Demand profiles and baseline demand tables. · `data/demand`
- [Network data](https://github.com/andrewlyden/pypsa-gb/tree/master/data/network): spatial source · ETYS, reduced-network, zonal, and boundary mapping inputs. · `data/network`
- [Renewables](https://github.com/andrewlyden/pypsa-gb/tree/master/data/renewables): source data · Renewable planning and marine/offshore site datasets. · `data/renewables`
- [Market data](https://github.com/andrewlyden/pypsa-gb/tree/master/data/market): market source · Elexon bid/offer records and calibration artifacts. · `data/market`
- [Interconnectors](https://github.com/andrewlyden/pypsa-gb/tree/master/data/interconnectors): source data · Registers, mappings, and flow-related inputs for cross-border assets. · `data/interconnectors`

### Preprocess

Domain ETL modules that transform raw datasets into model-ready assets.

- [Demand ETL](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/demand/integrate.py): demand integration · Builds demand profiles and flexibility additions. · `scripts/demand/integrate.py`
- [Gen ETL](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/generators/finalize_generator_integration.py): generator integration · Maps generators, enriches locations, and assigns capacities/costs. · `scripts/generators/finalize_generator_integration.py`
- [Renewables ETL](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/renewables/renewable_integration.py): renewables integration · Generates and maps renewable availability profiles to the network. · `scripts/renewables/renewable_integration.py`
- [Storage ETL](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/storage/add_storage.py): storage integration · Adds storage assets and parameters into the model. · `scripts/storage/add_storage.py`
- [Interconnector ETL](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/interconnectors/add_to_network.py): asset integration · Cleans, maps, and adds interconnectors to the network. · `scripts/interconnectors/add_to_network.py`
- [Market ETL](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/market/elexon_data.py): market calibration · Processes Elexon market data and validates balancing-market logic. · `scripts/market/elexon_data.py`
- [Hydrogen](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/hydrogen/add_hydrogen_system.py): system add-on · Adds hydrogen system representation to the network. · `scripts/hydrogen/add_hydrogen_system.py`

### Network

Core network construction and clustering boundary.

- [Build network](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/network_build/build_network.py): network assembly · Assembles the PyPSA network from ETYS and spatial mappings. · `scripts/network_build/build_network.py`
- [Cluster](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/network_clustering/cluster_network.py): network reduction · Aggregates components into reduced or zonal network forms. · `scripts/network_clustering/cluster_network.py`

### Solve

Optimization and final network preparation.

- [Solve](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/solve/solve_network.py): optimization · Builds the solvable network and runs the optimization. · `scripts/solve/solve_network.py`
- [Finalize](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/solve/finalize_network.py): postprocess · Applies final network constraints and output shaping. · `scripts/solve/finalize_network.py`

### Analysis

Validation, comparison, and result analysis.

- [Analyze](https://github.com/andrewlyden/pypsa-gb/blob/master/scripts/analysis/analyze_results.py): analysis · Analyzes solved networks, scenarios, and market outcomes. · `scripts/analysis/analyze_results.py`
- [Validate](https://github.com/andrewlyden/pypsa-gb/blob/master/tests/workflow/test_scenario_validation.py): validation · Scenario and historical validation workflows and checks. · `tests/workflow/test_scenario_validation.py`

### Docs

Documentation and tutorial workflows.

- [Docs](https://github.com/andrewlyden/pypsa-gb/blob/master/docs/source/index.rst): documentation · Sphinx/nbsphinx documentation and tutorials mirroring workflows. · `docs/source/index.rst`

## Connections

- Snakefile → Rules: orchestrates · Main workflow delegates execution to rule sets.
- Cutouts → Rules: supports · Cutout preparation feeds renewable and flexibility workflows.
- YAML config → Config: loads · Configuration files are interpreted by the loader.
- Config → Snakefile: routes · Scenario and year selection drive the pipeline path.
- FES → Config: scenario inputs · Future scenario assumptions parameterize model runs.
- Generators → Gen ETL: inputs · Raw generator registries and caches feed generator ETL.
- Demand → Demand ETL: inputs · Demand profiles are transformed into model demand series.
- Network data → Build network: inputs · Network topology and spatial boundaries feed assembly.
- Renewables → Renewables ETL: inputs · Renewable site and planning datasets feed profile generation.
- Market data → Market ETL: inputs · Elexon market records feed calibration and validation.
- Interconnectors → Interconnector ETL: inputs · Register and mapping data feed interconnector ETL.
- Demand ETL → Build network: injects · Demand loads are attached to the assembled network.
- Gen ETL → Build network: injects · Generators are mapped into the network model.
- Renewables ETL → Build network: injects · Renewable assets and profiles are attached to buses.
- Storage ETL → Build network: injects · Storage assets become part of the network.
- Interconnector ETL → Build network: injects · Cross-border assets are connected to the network.
- Hydrogen → Build network: injects · Hydrogen system components extend the network.
- Market ETL → Solve: calibrates · Market-derived parameters influence solving and constraints.
- Build network → Cluster: reduces · Built network can be clustered into reduced abstractions.
- Cluster → Solve: prepares · Clustered networks are handed to the solver.
- Build network → Solve: solves · The assembled network is the core optimization input.
- Solve → Finalize: outputs · Solved results are finalized for downstream use.
- Finalize → Analyze: analyzed by · Final outputs feed result analysis and comparison.
- Finalize → Validate: validated by · Solved networks are checked against historical and scenario tests.
- Analyze → Docs: documented in · Analysis workflows are mirrored in tutorials and docs.
- Validate → Docs: documented in · Validation workflows and tutorials are described in docs.
