# pypsa/pypsa-eur architecture

> Architecture diagram of the GitHub repository pypsa/pypsa-eur, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pypsa/pypsa-eur
- Repository: https://github.com/pypsa/pypsa-eur

## Overview

##### Project Type and Purpose
- The project is a large-scale energy system modelling tool designed to simulate and analyse the European transmission network. It supports both operational studies and long-term generation and transmission expansion planning.
- The model ingests real-world grid data (from ENTSO-E and GridKit), renewable and demand time series (from OPSD, ERA5, atlite), and also includes an extensive sector-coupled extension (covering transport, industry, heating, etc.).

##### Main Components
a. Data and Configuration:
– Data Directory: Represents various CSV, JSON, and YAML datasets (e.g., grid data, demand, potentials).
– Configuration Files: The “config” folder (with default YAML settings, scenario templates, and plotting configurations) that steer model parameters and scenario setups.

b. Build and Preprocessing Modules:
– Scripts in the “scripts” directory are responsible for multiple tasks such as building power plants, processing power and heat demand profiles, clustering network nodes, and preparing geographic as well as temporal data.
– The “rules” directory holds the Snakemake workflow scripts that orchestrate these tasks (like data retrieval, network building, and post-processing).

c. Solve (Optimisation) Modules:
– Although a custom note mentioned “i miss the solve scripts”, the project actually contains multiple solving routines:
- In the “scripts” folder: files such as “solve_network.py” and “solve_operations_network.py” perform the optimisation routines.
- In the “rules” folder: several Snakemake rules (e.g., “solve_electricity.smk”, “solve_myopic.smk”, “solve_overnight.smk”, “solve_perfect.smk”) coordinate the execution of different optimisation methods.
– These solving components are vital as they execute the network optimisation using the PyPSA framework.

d. Documentation, Testing, and Deployment:
– GitHub workflows, Docker files, and pre-commit configurations support CI/CD and development best practices.
– “test” directory contains various tests to validate components like the base network and building routines.

##### Relationships and Interactions
- Data flows start from raw data (in “data”) and configuration files (in “config”) into data processing scripts in “scripts” and Snakemake rules. These modules preprocess and cluster data for the model.
- Preprocessed data is then passed to the core network building components (also part of “scripts” and “rules”) which integrate with PyPSA as the central energy system model.
- The solve scripts (both stand-alone and via Snakemake rules) take the fully built network model and perform optimisation, determining optimal power flows and generation capacities.
- Postprocessing scripts generate plots, summaries, and tables that are eventually documented (in “doc”) and tested (in “test”).

##### Key Architectural Patterns and Design Principles
- Modular Design: Clear separation between data ingestion, model building, solution (optimisation) routines, and postprocessing/visualization.
- Workflow Orchestration: Use of Snakemake (found in the “rules” folder) to manage dependencies and steps, supporting reproducibility.
- Extensibility: Structured directory layout (e.g., separate “scripts” and “config” folders) that allows extending or modifying certain aspects (like adding new solving methods or data sources).
- Maintainability: The use of tests, CI workflows, and detailed documentation emphasize sustainable development practices.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Configuration Layer
    subgraph "Data & Configuration"
        DATA("Raw Data (CSV/JSON/YAML)"):::dataStyle
        CONFIG("Configurations (YAML)"):::dataStyle
    end

    %% Data Processing & Network Building Layer
    subgraph "Data Processing & Network Building"
        BUILD("Build Scripts (e.g., build_powerplants.py)"):::buildStyle
        RULES_BUILD("Snakemake Build Rules (e.g., build_electricity.smk)"):::buildStyle
    end

    %% Optimization / Solve Layer
    subgraph "Optimization / Solve"
        SOLVE("Solve Script (solve_network.py)"):::solveStyle
        SOLVE_OP("Operations Solve Script (solve_operations_network.py)"):::solveStyle
        RULES_SOLVE("Snakemake Solve Rules (e.g., solve_electricity.smk)"):::solveStyle
    end

    %% Postprocessing & Analysis Layer
    subgraph "Postprocessing & Analysis"
        DOC("Documentation & Analysis (doc, README.md)"):::docStyle
    end

    %% Testing & CI/CD Layer
    subgraph "Testing & CI/CD"
        TEST("Testing (test)"):::ciStyle
        CICD("CI/CD Workflows (GitHub, Docker, pre-commit)"):::ciStyle
    end

    %% Connections between layers
    DATA --> BUILD
    CONFIG --> BUILD
    BUILD --> RULES_BUILD
    RULES_BUILD --> SOLVE
    SOLVE --> RULES_SOLVE
    SOLVE_OP --> RULES_SOLVE
    RULES_SOLVE --> DOC
    DOC --> TEST
    DOC --> CICD

    %% Click Events
    click DATA "https://github.com/pypsa/pypsa-eur/tree/master/data"
    click CONFIG "https://github.com/pypsa/pypsa-eur/tree/master/config"
    click BUILD "https://github.com/pypsa/pypsa-eur/blob/master/scripts/build_powerplants.py"
    click RULES_BUILD "https://github.com/pypsa/pypsa-eur/blob/master/rules/build_electricity.smk"
    click SOLVE "https://github.com/pypsa/pypsa-eur/blob/master/scripts/solve_network.py"
    click SOLVE_OP "https://github.com/pypsa/pypsa-eur/blob/master/scripts/solve_operations_network.py"
    click RULES_SOLVE "https://github.com/pypsa/pypsa-eur/blob/master/rules/solve_electricity.smk"
    click DOC "https://github.com/pypsa/pypsa-eur/tree/master/doc"
    click TEST "https://github.com/pypsa/pypsa-eur/tree/master/test"
    click CICD "https://github.com/pypsa/pypsa-eur/tree/master/.github/workflows"

    %% Styles
    classDef dataStyle fill:#a8ddb5,stroke:#333,stroke-width:2px;
    classDef buildStyle fill:#fc8d62,stroke:#333,stroke-width:2px;
    classDef solveStyle fill:#66c2a5,stroke:#333,stroke-width:2px;
    classDef docStyle fill:#8da0cb,stroke:#333,stroke-width:2px;
    classDef ciStyle fill:#e78ac3,stroke:#333,stroke-width:2px;
```
