# doi-usgs/nhm-assist architecture

> Architecture diagram of the GitHub repository doi-usgs/nhm-assist, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/doi-usgs/nhm-assist
- Repository: https://github.com/doi-usgs/nhm-assist

## Overview

1. **What this repo is**
   - `nhm-assist` is a notebook-centered hydrologic analysis toolkit for NHM subdomain models.
   - The core user-facing outputs are generated notebooks, interactive HTML maps, and interactive plots for inspecting model structure, parameters, simulations, and gage comparisons.

2. **Top-level architecture**
   - The repository is organized into three workflow families:
     - **NHM**: run and analyze NHM subdomain models with `pywatershed`
     - **NHF**: hydrofabric-oriented inspection and visualization
     - **PEST / PEST++ IES calibration**: calibration support, forward runs, and post-processing
   - These workflows share a common pattern: Python scripts are converted into notebooks and then executed interactively.

3. **Source package boundary**
   - The canonical importable code lives under `src/assist`.
   - It is split into namespaces:
     - `assist.nhm` for NHM workflow utilities
     - `assist.nhf` for hydrofabric-specific utilities
     - `assist.pest` for PEST helpers
   - This `src/` layout is the main code boundary for packaging and reuse.

4. **Workflow template boundary**
   - `src/workflow_templates` contains the notebook source scripts that users generate into `.ipynb` files.
   - There are separate template trees for `nhm`, `nhf`, and `pest`.
   - These scripts are not the runtime library; they are the workflow entrypoints that orchestrate reading data, calling helpers, and writing notebook artifacts.

5. **Notebook generation flow**
   - `make_notebooks.py` is the central build step that transforms workflow `.py` files into notebooks.
   - The same concept exists inside the workflow-specific folders, including `pestpp_ies_calibration/make_notebooks.py` and `pestpp_ies_calibration/helpers/make_notebooks.py`-style support.
   - This means the repo’s primary execution model is “generate notebooks first, then run them,” rather than shipping a single application binary.

6. **NHM / NHF analysis subsystem**
   - The `nhm` and `nhf` helper modules implement geospatial and model-interpretation functionality.
   - These helpers support:
     - hydrofabric inspection
     - HRU-to-segment connectivity checks
     - parameter mapping and visualization
     - streamflow observation handling
     - output visualization for variables like recharge or evapotranspiration
   - The implementation relies heavily on geospatial data handling plus plotting/mapping libraries.

7. **Visualization output subsystem**
   - Interactive plots are built with `plotly`, and maps are built with `folium`.
   - Generated artifacts are written to notebook output folders as HTML, so they are decoupled from Jupyter and can be opened in a browser or embedded elsewhere.
   - This output boundary is important: notebooks produce portable HTML deliverables, not just inline notebook cells.

8. **Calibration subsystem**
   - `pestpp_ies_calibration` is a distinct calibration workspace, not just a helper module.
   - It includes:
     - calibration templates and ancillary CSVs
     - forward-run and post-processing scripts
     - hotstart/result transfer utilities
     - Condor-related preparation scripts for distributed execution
   - The calibration flow is centered on preparing PEST/PEST++ IES inputs, executing model runs, and consolidating results.

9. **Vendored dependency boundary**
   - The PEST workflow vendors a local copy of `pyemu` under `pestpp_ies_calibration/dependencies/pyemu`.
   - This suggests the calibration workflow is designed to be self-contained and resilient to upstream dependency changes.
   - In contrast, the rest of the repo uses the shared environment in `environment.yaml`.

10. **Domain data and static inputs**
   - `domain_data/` holds example or user-provided NHM subdomains, including a concrete `Walla_Walla` example.
   - This area contains model control files, NetCDF outputs, geopackages, parameter files, and observation data.
   - `data_dependencies/` stores reusable static inputs such as shapefiles, CSV lookup tables, images, and explanation tables used by notebooks.

11. **Data flow through the system**
   - External or provided model data is pulled into `domain_data/` using `pull_domain.py`.
   - Workflow notebooks read from `domain_data/` and `data_dependencies/`, then call helper modules from `src/assist` or legacy mirrors.
   - Outputs are written back into domain-specific notebook output folders as plots, maps, and calibration artifacts.

12. **Legacy/backward-compatibility layout**
   - There are top-level copies of older helper packages such as `nhm_helpers/`, `nhf_assist/`, and `pestpp_ies_calibration/helpers/`.
   - These appear to support older notebook paths and import conventions alongside the newer `src/assist` package layout.
   - For architecture diagrams, treat `src/assist` as the current library core and the top-level helper directories as compatibility mirrors or workflow-local support code.

13. **Automation and testing**
   - GitHub Actions CI lives under `.github/workflows`, with helper scripts for notebook testing and cache management.
   - The CI setup indicates notebooks are part of the test surface, so repository validation likely includes generating notebooks and running selected notebook checks.
   - Pre-commit and packaging metadata (`pyproject.toml`, `environment.yaml`) reinforce that this is a reproducible Python workflow repository, not just a notebook dump.

14. **Main external/runtime dependencies**
   - The runtime is Python in a conda/mamba environment.
   - Key external libraries shaping the architecture are `pywatershed` for NHM simulation, `plotly` and `folium` for visualization, `pyemu`/PEST++ for calibration, and geospatial tooling for shapefile/geopackage handling.
   - The repository also references HyTEST OSN storage as the source for downloadable example domains.

Main entry points: [`.github/workflows/ci.yaml`](https://github.com/doi-usgs/nhm-assist/blob/main/.github/workflows/ci.yaml) (CI notebooks), [`src/workflow_templates/make_notebooks.py`](https://github.com/doi-usgs/nhm-assist/blob/main/src/workflow_templates/make_notebooks.py) (make notebooks), [`src/assist/nhm`](https://github.com/doi-usgs/nhm-assist/tree/main/src/assist/nhm) (nhm), [`src/assist/nhf`](https://github.com/doi-usgs/nhm-assist/tree/main/src/assist/nhf) (nhf)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_source["Source package"]
  node_node_src_assist["assist<br/>python package"]
  node_node_src_nhm["nhm<br/>analysis helpers"]
  node_node_src_nhf["nhf<br/>hydrofabric helpers"]
  node_node_src_pest["pest<br/>calibration helpers"]
end

subgraph group_grp_workflows["Workflow templates"]
  node_node_tpl_make["make notebooks<br/>build script<br/>[make_notebooks.py]"]
  node_node_tpl_nhm["NHM templates<br/>workflow scripts"]
  node_node_tpl_nhf["NHF templates<br/>workflow scripts"]
  node_node_tpl_pest["PEST templates<br/>workflow scripts"]
end

subgraph group_grp_calibration["Calibration workspace"]
  node_node_cal_root["calibration ws<br/>calibration workspace"]
  node_node_cal_helpers["calibration helpers<br/>workflow support"]
  node_node_cal_pyemu["vendored pyemu<br/>vendored dependency"]
end

subgraph group_grp_data["Data and examples"]
  node_node_domain[("domain data<br/>example domains")]
  node_node_dd[("data deps<br/>static inputs")]
  node_node_pull["pull domain<br/>ingest script<br/>[pull_domain.py]"]
end

subgraph group_grp_automation["Automation"]
  node_node_ci["CI notebooks<br/>automation<br/>[ci.yaml]"]
  node_node_ci_scripts["CI scripts<br/>test helpers<br/>[scripts]"]
end

node_node_tpl_make -->|"builds"| node_node_tpl_nhm
node_node_tpl_make -->|"builds"| node_node_tpl_nhf
node_node_tpl_make -->|"builds"| node_node_tpl_pest
node_node_tpl_nhm -->|"uses"| node_node_src_nhm
node_node_tpl_nhf -->|"uses"| node_node_src_nhf
node_node_tpl_pest -->|"uses"| node_node_src_pest
node_node_src_nhm -->|"reads"| node_node_domain
node_node_src_nhf -->|"reads"| node_node_domain
node_node_src_nhm -->|"reads"| node_node_dd
node_node_src_nhf -->|"reads"| node_node_dd
node_node_tpl_pest -->|"orchestrates"| node_node_cal_root
node_node_cal_root -->|"delegates"| node_node_cal_helpers
node_node_cal_helpers -->|"depends on"| node_node_cal_pyemu
node_node_cal_root -->|"writes"| node_node_domain
node_node_pull -->|"stages"| node_node_domain
node_node_ci -->|"executes"| node_node_ci_scripts
node_node_ci -->|"validates"| node_node_tpl_make
node_node_ci -->|"tests"| node_node_tpl_nhm
node_node_ci -->|"tests"| node_node_tpl_nhf
node_node_ci -->|"tests"| node_node_tpl_pest
node_node_src_assist -->|"contains"| node_node_src_nhm
node_node_src_assist -->|"contains"| node_node_src_nhf
node_node_src_assist -->|"contains"| node_node_src_pest

click node_node_src_assist "https://github.com/doi-usgs/nhm-assist/tree/main/src/assist"
click node_node_src_nhm "https://github.com/doi-usgs/nhm-assist/tree/main/src/assist/nhm"
click node_node_src_nhf "https://github.com/doi-usgs/nhm-assist/tree/main/src/assist/nhf"
click node_node_src_pest "https://github.com/doi-usgs/nhm-assist/tree/main/src/assist/pest"
click node_node_tpl_make "https://github.com/doi-usgs/nhm-assist/blob/main/src/workflow_templates/make_notebooks.py"
click node_node_tpl_nhm "https://github.com/doi-usgs/nhm-assist/tree/main/src/workflow_templates/nhm"
click node_node_tpl_nhf "https://github.com/doi-usgs/nhm-assist/tree/main/src/workflow_templates/nhf"
click node_node_tpl_pest "https://github.com/doi-usgs/nhm-assist/tree/main/src/workflow_templates/pest"
click node_node_cal_root "https://github.com/doi-usgs/nhm-assist/tree/main/pestpp_ies_calibration"
click node_node_cal_helpers "https://github.com/doi-usgs/nhm-assist/tree/main/pestpp_ies_calibration/helpers"
click node_node_cal_pyemu "https://github.com/doi-usgs/nhm-assist/tree/main/pestpp_ies_calibration/dependencies/pyemu"
click node_node_domain "https://github.com/doi-usgs/nhm-assist/tree/main/domain_data"
click node_node_dd "https://github.com/doi-usgs/nhm-assist/tree/main/data_dependencies"
click node_node_pull "https://github.com/doi-usgs/nhm-assist/blob/main/pull_domain.py"
click node_node_ci "https://github.com/doi-usgs/nhm-assist/blob/main/.github/workflows/ci.yaml"
click node_node_ci_scripts "https://github.com/doi-usgs/nhm-assist/blob/main/.github/scripts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_src_assist,node_node_src_nhm,node_node_src_nhf,node_node_src_pest toneBlue
class node_node_tpl_make,node_node_tpl_nhm,node_node_tpl_nhf,node_node_tpl_pest toneAmber
class node_node_cal_root,node_node_cal_helpers,node_node_cal_pyemu toneMint
class node_node_domain,node_node_dd,node_node_pull toneRose
class node_node_ci,node_node_ci_scripts toneIndigo
```

## Components

### Source package

Current importable library code under src/assist.

- [assist](https://github.com/doi-usgs/nhm-assist/tree/main/src/assist): python package · Canonical reusable library boundary for the repository. · `src/assist`
- [nhm](https://github.com/doi-usgs/nhm-assist/tree/main/src/assist/nhm): analysis helpers · NHM-oriented utilities for model inspection, plotting, and pywatershed workflows. · `src/assist/nhm`
- [nhf](https://github.com/doi-usgs/nhm-assist/tree/main/src/assist/nhf): hydrofabric helpers · Hydrofabric-specific utilities, visualization, and geospatial model inspection. · `src/assist/nhf`
- [pest](https://github.com/doi-usgs/nhm-assist/tree/main/src/assist/pest): calibration helpers · Shared PEST utilities used by calibration workflows. · `src/assist/pest`

### Workflow templates

Notebook source scripts that are converted into runnable notebooks.

- [make notebooks](https://github.com/doi-usgs/nhm-assist/blob/main/src/workflow_templates/make_notebooks.py): build script · Converts workflow source scripts into notebook artifacts. · `src/workflow_templates/make_notebooks.py`
- [NHM templates](https://github.com/doi-usgs/nhm-assist/tree/main/src/workflow_templates/nhm): workflow scripts · Notebook source scripts for NHM analysis and model runs. · `src/workflow_templates/nhm`
- [NHF templates](https://github.com/doi-usgs/nhm-assist/tree/main/src/workflow_templates/nhf): workflow scripts · Notebook source scripts for hydrofabric inspection and visualization. · `src/workflow_templates/nhf`
- [PEST templates](https://github.com/doi-usgs/nhm-assist/tree/main/src/workflow_templates/pest): workflow scripts · Notebook source scripts for PEST and PEST++ IES preparation and forward runs. · `src/workflow_templates/pest`

### Calibration workspace

PEST and PEST++ IES calibration preparation, execution, and post-processing.

- [calibration ws](https://github.com/doi-usgs/nhm-assist/tree/main/pestpp_ies_calibration): calibration workspace · Standalone workspace for PEST++ IES calibration and result handling. · `pestpp_ies_calibration`
- [calibration helpers](https://github.com/doi-usgs/nhm-assist/tree/main/pestpp_ies_calibration/helpers): workflow support · Forward runs, post-processing, Condor prep, hotstart transfer, and metrics. · `pestpp_ies_calibration/helpers`
- [vendored pyemu](https://github.com/doi-usgs/nhm-assist/tree/main/pestpp_ies_calibration/dependencies/pyemu): vendored dependency · Local copy of pyemu bundled to keep calibration workflows self-contained. · `pestpp_ies_calibration/dependencies/pyemu`

### Data and examples

Reusable static inputs and example domain data.

- [domain data](https://github.com/doi-usgs/nhm-assist/tree/main/domain_data): example domains · Example and user-provided NHM subdomain inputs and model outputs. · `domain_data`
- [data deps](https://github.com/doi-usgs/nhm-assist/tree/main/data_dependencies): static inputs · Reusable shapefiles, lookup tables, images, and explanation tables. · `data_dependencies`
- [pull domain](https://github.com/doi-usgs/nhm-assist/blob/main/pull_domain.py): ingest script · Fetches or stages example domain data into domain_data. · `pull_domain.py`

### Automation

Notebook generation and CI support.

- [CI notebooks](https://github.com/doi-usgs/nhm-assist/blob/main/.github/workflows/ci.yaml): automation · Notebook checks and cache management used by GitHub Actions. · `.github/workflows/ci.yaml`
- [CI scripts](https://github.com/doi-usgs/nhm-assist/blob/main/.github/scripts): test helpers · Scheduled-test cache handling and notebook execution helpers. · `.github/scripts`

## Connections

- make notebooks → NHM templates: builds · Generates notebook artifacts from NHM workflow scripts.
- make notebooks → NHF templates: builds · Generates notebook artifacts from NHF workflow scripts.
- make notebooks → PEST templates: builds · Generates notebook artifacts from PEST workflow scripts.
- NHM templates → nhm: uses · NHM templates orchestrate reusable NHM helpers.
- NHF templates → nhf: uses · NHF templates orchestrate reusable NHF helpers.
- PEST templates → pest: uses · PEST templates call shared calibration utilities.
- nhm → domain data: reads · NHM analysis consumes subdomain controls, outputs, and observations.
- nhf → domain data: reads · Hydrofabric visualization consumes model GIS and domain outputs.
- nhm → data deps: reads · NHM helpers use lookup tables, shapefiles, and explanation assets.
- nhf → data deps: reads · NHF helpers use static geospatial inputs and map assets.
- PEST templates → calibration ws: orchestrates · PEST notebooks set up and run the calibration workspace.
- calibration ws → calibration helpers: delegates · Calibration workspace uses helper scripts for run and post-processing stages.
- calibration helpers → vendored pyemu: depends on · Calibration support uses the vendored pyemu implementation.
- calibration ws → domain data: writes · Calibration runs read and update domain-specific run folders and artifacts.
- pull domain → domain data: stages · Pulls example model data into the domain_data workspace.
- CI notebooks → CI scripts: executes · CI uses helper scripts to test notebooks and manage cached outputs.
- CI notebooks → make notebooks: validates · CI checks notebook generation and notebook-based workflows.
- CI notebooks → NHM templates: tests · Notebook workflows are part of the validation surface.
- CI notebooks → NHF templates: tests · Notebook workflows are part of the validation surface.
- CI notebooks → PEST templates: tests · Notebook workflows are part of the validation surface.
- assist → nhm: contains · Top-level package exposes NHM namespace.
- assist → nhf: contains · Top-level package exposes NHF namespace.
- assist → pest: contains · Top-level package exposes PEST namespace.