# pypsa-meets-earth/pypsa-earth architecture

> Architecture diagram of the GitHub repository pypsa-meets-earth/pypsa-earth, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pypsa-meets-earth/pypsa-earth
- Repository: https://github.com/pypsa-meets-earth/pypsa-earth

## Overview

##### Overall Project Type and Purpose
- Recognize that PyPSA-Earth is not a traditional client–server application but a computational simulation and optimization tool.
- Its core objective is to provide a flexible, data-driven, multi-sector energy system model that can be customized and extended for global or regional studies.
- The provided README describes its integration with the PyPSA framework and emphasis on cross-sector coupling (electricity, heating, transport, etc.) with high spatial and temporal resolution.

##### Analyzing the File Structure for Architectural Components
- Top-level directories like “configs”, “data”, “doc”, “scripts”, and “test” indicate a modular separation:
- The “configs” folder holds YAML configuration files (including domain-specific ones like “powerplantmatching_config.yaml” and “scenarios”) which direct how the simulation runs.
- The “data” directory contains CSV files and input data sets—representing raw, preprocessed, and custom simulation inputs.
- The “scripts” folder is populated with numerous Python scripts that handle different parts of data preprocessing, model building, network simulation (e.g., “build_powerplants.py”, “prepare_network.py”, “solve_network.py”), and postprocessing (e.g., plotting and summary generation).
- The “Snakefile” and files in “test” suggest an orchestrated workflow using Snakemake for reproducible model runs.
- Development and deployment are supported by containerization (.devcontainer, Dockerfile) and CI/CD configurations (.github/workflows), which help maintain consistency across environments.

##### Key Components
a. Data Ingestion and Configuration:
- “data” directory as the primary input module (comprising CSVs, custom data folders, etc.).
- “configs” as the configuration management layer that drives simulation parameters and scenario settings.

b. Data Preprocessing and Preparation:

c. Workflow Orchestration:
- A “Workflow Engine” module represented by the “Snakefile” and Snakemake workflow, which sequentially invokes the various processing and simulation scripts based on dependencies.

d. Core Optimization / Simulation Engine:
- “Model Building & Optimisation” component. This is the heart of the system where the prepared data is fed into the PyPSA framework to build network models (using scripts like “prepare_network.py”, “solve_network.py”, “monte_carlo.py”).
- Integration with external solvers (e.g., the HiGHs solver) and dependencies managed via Conda environments.

e. Output Generation and Post-Processing:
- Optionally, reference the “doc” folder that contains documentation and examples showing how results are interpreted and disseminated.

f. Supporting Components:
- Testing and Validation: A separate component for tests (using files in the “test” folder) confirms the integrity of the model and simulation outputs.

##### Illustrating Relationships and Interactions
- Inputs (data and configuration files) feed into preprocessing scripts.
- Preprocessed outputs then pass to the model building and optimization engine.
- The workflow engine (controlled via Snakemake) orchestrates these processes.
- Post-processing scripts consume the simulation results and drive visualization and reporting.
- Clearly delineate how configuration settings and custom scenario files influence various parts of the data pipeline and simulation.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data & Configuration Ingestion
    subgraph "Data & Config Ingestion"
        A1["Data Directory"]:::input
        A2["Configuration Files"]:::input
    end

    %% Data Preprocessing and Preparation
    subgraph "Data Preprocessing"
        B1["Preprocessing Scripts"]:::processing
    end

    %% Workflow Orchestration
    subgraph "Workflow Engine"
        C1["Snakefile"]:::orchestration
        C2["GitHub Workflows"]:::orchestration
        C3["Test Directory"]:::orchestration
    end

    %% Core Optimization / Simulation Engine
    subgraph "Core Optimization"
        D1["Simulation & Optimization Scripts"]:::simulation
    end

    %% Output Generation and Post-Processing
    subgraph "Output Generation"
        E1["Result Processing & Visualization"]:::postprocessing
        E2["Documentation"]:::postprocessing
    end

    %% Supporting Development & Deployment
    subgraph "Supporting Tools"
        F1["Container & Env Config"]:::support
        F2["CI/CD & Testing"]:::support
    end

    %% Arrows and Data Flow
    A1 ---|"raw data"| B1
    A2 ---|"config parameters"| B1
    B1 ---|"prepared data"| D1
    C1 ---|"orchestrates"| B1
    C1 ---|"orchestrates"| D1
    D1 ---|"simulation results"| E1
    D1 ---|"results summary"| E2

    %% Supporting tools integration
    F1 ---|"deployment setup"| C1
    F2 ---|"CI/CD triggers"| C2
    F2 ---|"testing workflows"| C3

    %% Click Events
    click A1 "https://github.com/pypsa-meets-earth/pypsa-earth/tree/main/data"
    click A2 "https://github.com/pypsa-meets-earth/pypsa-earth/tree/main/configs"
    click B1 "https://github.com/pypsa-meets-earth/pypsa-earth/tree/main/scripts"
    click C1 "https://github.com/pypsa-meets-earth/pypsa-earth/tree/main/Snakefile"
    click C2 "https://github.com/pypsa-meets-earth/pypsa-earth/tree/main/.github/workflows"
    click C3 "https://github.com/pypsa-meets-earth/pypsa-earth/tree/main/test"
    click D1 "https://github.com/pypsa-meets-earth/pypsa-earth/tree/main/scripts"
    click E1 "https://github.com/pypsa-meets-earth/pypsa-earth/tree/main/scripts"
    click E2 "https://github.com/pypsa-meets-earth/pypsa-earth/tree/main/doc"
    click F1 "https://github.com/pypsa-meets-earth/pypsa-earth/blob/main/.devcontainer"
    click F2 "https://github.com/pypsa-meets-earth/pypsa-earth/tree/main/.github/workflows"

    %% Styles
    classDef input fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef processing fill:#A9DFBF,stroke:#1E8449,stroke-width:2px;
    classDef orchestration fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef simulation fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef postprocessing fill:#F5B7B1,stroke:#922B21,stroke-width:2px;
    classDef support fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;
```
