# pypsa/pypsa-usa architecture

> Architecture diagram of the GitHub repository pypsa/pypsa-usa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pypsa/pypsa-usa
- Repository: https://github.com/pypsa/pypsa-usa

## Overview

##### Project Type and Purpose
- PyPSA-USA is an open-source workflow for building, solving, and analyzing a bulk transmission energy model of the U.S. It is not a classic web app but a data-driven modeling pipeline using Snakemake to orchestrate Python scripts and external datasets.

##### File Structure Analysis
- Top level:
- Configuration & CI: `.github/`, `.pre-commit-config.yaml`, `pyproject.toml`, `uv.lock`
- Documentation: `docs/`, `README.md`, `CITATION.cff`, `.readthedocs.yaml`
- Workflow entrypoint: `workflow/` (Snakemake rules, config, environments, notebooks, repo_data, scripts)
- Shell helpers: `init_pypsa_usa.sh`, `run_slurm.sh`, `.test_sh`

- Within `workflow/`:
- `repo_data/`: cached static datasets (EIA, ReEDS, shapefiles, cost tables, etc.)
- `config/`: Snakemake/conda environment YAMLs, policy constraint CSVs
- `rules/`: Snakemake rule definitions (`build_*.smk`, `solve_*.smk`, etc.)
- `scripts/`: Python modules for each pipeline stage (data retrieval, build network, clustering, solve, summary, plotting)
- `notebooks/`: illustrative Jupyter notebooks for cutouts, industry, validation, etc.
- `snakemake_profiles/`: Slurm profile for HPC execution

##### Key README Insights
- Describes the project as an energy-system optimization framework based on PyPSA.
- Use cases: capacity expansion, production cost simulation, power-flow analysis.

a. Main Components
1. External Data Sources (EIA, PUDL, grid emissions API, ReEDS constraints, shapefiles)
2. Local Data Store (`workflow/repo_data`)
3. Configuration Layer (`workflow/config`, `pyproject.toml`, `uv.lock`)
4. Orchestration Engine (Snakemake + Slurm profile)
5. Processing Scripts (`workflow/scripts/*.py`)
– Data Retrieval
– Data Cleaning & Preprocessing
– Network Building (PyPSA network objects)
– Clustering & Region Mapping
– Model Solve (capacity expansion, dispatch)
– Post-processing & Summaries
– Plotting & Visualization
6. Core Modeling Library (PyPSA)
7. Outputs (network files, CSV summaries, plots)
8. Documentation & Notebooks (`docs/`, `workflow/notebooks/`)
9. Continuous Integration (GitHub Actions)

b. Relationships & Data Flow
- Config files feed into Snakemake → determine which rules/scripts run
- Snakemake orchestrates sequential execution: Retrieval → Build → Solve → Postprocess → Plot
- Processing scripts invoke the PyPSA library for network creation and solving
- Outputs persist to disk and are referenced by documentation and notebooks

c. Architectural Patterns & Principles
- Data-centric pipeline (Snakemake as a Directed Acyclic Graph of tasks)
- Separation of concerns: each script handles a single domain (e.g., electricity, natural gas, sectors)
- Reproducibility via conda environments (`workflow/envs/*.yaml`) and lock files (`uv.lock`)
- Extensibility: easily add new Snakemake rules or scripts for additional data sources or analyses

d. Key Technologies
- Python + PyPSA for modeling
- Snakemake for workflow orchestration
- Conda/venv for environment management
- Jupyter Notebooks for exploratory analysis
- GitHub Actions for CI

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Data Sources
    subgraph "External Data Sources" 
        direction TB
        EDS["External Data Sources"]:::data
    end

    %% Local Data Store
    LDS["Local Data Store\n(workflow/repo_data/)"]:::data
    click LDS "https://github.com/pypsa/pypsa-usa/tree/master/workflow/repo_data/"

    %% Configuration Layer
    subgraph "Configuration Layer" 
        direction TB
        CFG1["workflow/config/"]:::config
        click CFG1 "https://github.com/pypsa/pypsa-usa/tree/master/workflow/config/"
        CFG2["workflow/envs/dev.yaml"]:::config
        click CFG2 "https://github.com/pypsa/pypsa-usa/blob/master/workflow/envs/dev.yaml"
        CFG3["workflow/envs/environment.yaml"]:::config
        click CFG3 "https://github.com/pypsa/pypsa-usa/blob/master/workflow/envs/environment.yaml"
        CFG4["pyproject.toml"]:::config
        click CFG4 "https://github.com/pypsa/pypsa-usa/blob/master/pyproject.toml"
        CFG5["uv.lock"]:::config
        click CFG5 "https://github.com/pypsa/pypsa-usa/blob/master/uv.lock"
    end

    %% Orchestration Engine
    subgraph "Orchestration Engine" 
        direction TB
        ORC1["workflow/Snakefile"]:::orchestration
        click ORC1 "https://github.com/pypsa/pypsa-usa/tree/master/workflow/Snakefile"
        ORC2["workflow/rules/"]:::orchestration
        click ORC2 "https://github.com/pypsa/pypsa-usa/tree/master/workflow/rules/"
        ORC3["workflow/run_slurm.sh"]:::orchestration
        click ORC3 "https://github.com/pypsa/pypsa-usa/blob/master/workflow/run_slurm.sh"
        ORC4["workflow/snakemake_profiles/slurm/config.yaml"]:::orchestration
        click ORC4 "https://github.com/pypsa/pypsa-usa/blob/master/workflow/snakemake_profiles/slurm/config.yaml"
    end

    %% Processing Scripts
    subgraph "Processing Scripts" 
        direction TB
        subgraph "Data Retrieval" 
            direction TB
            DR["Data Retrieval Scripts\n(workflow/scripts/)"]:::processing
            click DR "https://github.com/pypsa/pypsa-usa/tree/master/workflow/scripts/"
        end
        subgraph "Data Cleaning & Preprocessing"
            direction TB
            DC["Cleaning & Preprocessing"]:::processing
            click DC "https://github.com/pypsa/pypsa-usa/tree/master/workflow/scripts/"
        end
        subgraph "Network Building"
            direction TB
            NB["Network Building"]:::processing
            click NB "https://github.com/pypsa/pypsa-usa/tree/master/workflow/scripts/"
        end
        subgraph "Clustering & Region Mapping"
            direction TB
            CL["Clustering & Region Mapping"]:::processing
            click CL "https://github.com/pypsa/pypsa-usa/tree/master/workflow/scripts/"
        end
        subgraph "Model Solve"
            direction TB
            MS["solve_network.py"]:::processing
            click MS "https://github.com/pypsa/pypsa-usa/blob/master/workflow/scripts/solve_network.py"
        end
        subgraph "Post-processing & Summaries"
            direction TB
            PP["Post-processing & Summaries"]:::processing
            click PP "https://github.com/pypsa/pypsa-usa/tree/master/workflow/scripts/"
        end
        subgraph "Plotting & Visualization"
            direction TB
            PV["Plotting & Visualization"]:::processing
            click PV "https://github.com/pypsa/pypsa-usa/tree/master/workflow/scripts/"
        end
    end

    %% Core Modeling Library
    PyPSA["Core Modeling Library\n(PyPSA)"]:::model

    %% Outputs
    OUT["Outputs\n(network files, CSV, plots)"]:::output

    %% Documentation & Notebooks
    subgraph "Documentation & Notebooks"
        direction TB
        DOC["docs/"]:::docs
        click DOC "https://github.com/pypsa/pypsa-usa/tree/master/docs/"
        NBK["workflow/notebooks/"]:::docs
        click NBK "https://github.com/pypsa/pypsa-usa/tree/master/workflow/notebooks/"
    end

    %% Continuous Integration
    subgraph "Continuous Integration"
        direction TB
        CI1[".github/workflows/main.yml"]:::ci
        click CI1 "https://github.com/pypsa/pypsa-usa/blob/master/.github/workflows/main.yml"
        CI2[".pre-commit-config.yaml"]:::ci
        click CI2 "https://github.com/pypsa/pypsa-usa/blob/master/.pre-commit-config.yaml"
    end

    %% Data Flow
    EDS -->|download| LDS
    LDS -->|read| DR
    CFG1 & CFG2 & CFG3 & CFG4 & CFG5 --> ORC1
    ORC1 --> ORC2
    ORC1 --> ORC3
    ORC1 --> ORC4
    ORC1 --> DR
    ORC1 --> DC
    ORC1 --> NB
    ORC1 --> CL
    ORC1 --> MS
    ORC1 --> PP
    ORC1 --> PV

    DR --> DC
    DC --> NB
    NB --> CL
    CL --> MS
    MS --> PP
    PP --> PV
    PV --> OUT

    %% Scripts invoke PyPSA
    NB --> PyPSA
    MS --> PyPSA

    %% Outputs used by docs
    OUT --> DOC
    OUT --> NBK

    %% CI monitors repository
    CI1 --> ORC1
    CI2 --> ORC1

    %% Styles
    classDef data fill:#BBDEFB,stroke:#0D47A1;
    classDef config fill:#C8E6C9,stroke:#1B5E20;
    classDef orchestration fill:#FFE0B2,stroke:#E65100;
    classDef processing fill:#FFF9C4,stroke:#F57F17;
    classDef model fill:#E1BEE7,stroke:#4A148C;
    classDef output fill:#FFE0B2,stroke:#F57F17;
    classDef docs fill:#ECEFF1,stroke:#37474F;
    classDef ci fill:#ECEFF1,stroke:#37474F;
```
