How pypsa/pypsa-usa works
Project Type and Purpose
• PyPSA-USA is an open-source workflow for building, solving, and analyzing a bulk transmission energy model of the U.S. It is not a classic web app but a data-driven modeling pipeline using Snakemake to orchestrate Python scripts and external datasets.
File Structure Analysis
• Top level:
• Configuration & CI: .github/, .pre-commit-config.yaml, pyproject.toml, uv.lock
• Documentation: docs/, README.md, CITATION.cff, .readthedocs.yaml
• Workflow entrypoint: workflow/ (Snakemake rules, config, environments, notebooks, repo_data, scripts)
• Shell helpers: init_pypsa_usa.sh, run_slurm.sh, .test_sh
• Within workflow/:
• repo_data/: cached static datasets (EIA, ReEDS, shapefiles, cost tables, etc.)
• config/: Snakemake/conda environment YAMLs, policy constraint CSVs
• rules/: Snakemake rule definitions (build_*.smk, solve_*.smk, etc.)
• scripts/: Python modules for each pipeline stage (data retrieval, build network, clustering, solve, summary, plotting)
• notebooks/: illustrative Jupyter notebooks for cutouts, industry, validation, etc.
• snakemake_profiles/: Slurm profile for HPC execution
Key README Insights
• Describes the project as an energy-system optimization framework based on PyPSA.
• Use cases: capacity expansion, production cost simulation, power-flow analysis.
a. Main Components
1. External Data Sources (EIA, PUDL, grid emissions API, ReEDS constraints, shapefiles)
2. Local Data Store (workflow/repo_data)
3. Configuration Layer (workflow/config, pyproject.toml, uv.lock)
4. Orchestration Engine (Snakemake + Slurm profile)
5. Processing Scripts (workflow/scripts/*.py)
– Data Retrieval
– Data Cleaning & Preprocessing
– Network Building (PyPSA network objects)
– Clustering & Region Mapping
– Model Solve (capacity expansion, dispatch)
– Post-processing & Summaries
– Plotting & Visualization
6. Core Modeling Library (PyPSA)
7. Outputs (network files, CSV summaries, plots)
8. Documentation & Notebooks (docs/, workflow/notebooks/)
9. Continuous Integration (GitHub Actions)
b. Relationships & Data Flow
• Config files feed into Snakemake → determine which rules/scripts run
• Snakemake orchestrates sequential execution: Retrieval → Build → Solve → Postprocess → Plot
• Processing scripts invoke the PyPSA library for network creation and solving
• Outputs persist to disk and are referenced by documentation and notebooks
c. Architectural Patterns & Principles
• Data-centric pipeline (Snakemake as a Directed Acyclic Graph of tasks)
• Separation of concerns: each script handles a single domain (e.g., electricity, natural gas, sectors)
• Reproducibility via conda environments (workflow/envs/*.yaml) and lock files (uv.lock)
• Extensibility: easily add new Snakemake rules or scripts for additional data sources or analyses
d. Key Technologies
• Python + PyPSA for modeling
• Snakemake for workflow orchestration
• Conda/venv for environment management
• Jupyter Notebooks for exploratory analysis
• GitHub Actions for CI