doi-usgs/nhm-assist

1. What this repo is

- nhm-assist is a notebook-centered hydrologic analysis toolkit for NHM subdomain models.

- The core user-facing outputs are generated notebooks, interactive HTML maps, and interactive plots for inspecting model structure, parameters, simulations, and gage comparisons.

2. Top-level architecture

- The repository is organized into three workflow families:

- NHM: run and analyze NHM subdomain models with pywatershed

- NHF: hydrofabric-oriented inspection and visualization

- PEST / PEST++ IES calibration: calibration support, forward runs, and post-processing

- These workflows share a common pattern: Python scripts are converted into notebooks and then executed interactively.

3. Source package boundary

- The canonical importable code lives under src/assist.

- It is split into namespaces:

- assist.nhm for NHM workflow utilities

- assist.nhf for hydrofabric-specific utilities

- assist.pest for PEST helpers

- This src/ layout is the main code boundary for packaging and reuse.

4. Workflow template boundary

- src/workflow_templates contains the notebook source scripts that users generate into .ipynb files.

- There are separate template trees for nhm, nhf, and pest.

- These scripts are not the runtime library; they are the workflow entrypoints that orchestrate reading data, calling helpers, and writing notebook artifacts.

5. Notebook generation flow

- make_notebooks.py is the central build step that transforms workflow .py files into notebooks.

- The same concept exists inside the workflow-specific folders, including pestpp_ies_calibration/make_notebooks.py and pestpp_ies_calibration/helpers/make_notebooks.py-style support.

- This means the repo’s primary execution model is “generate notebooks first, then run them,” rather than shipping a single application binary.

6. NHM / NHF analysis subsystem

- The nhm and nhf helper modules implement geospatial and model-interpretation functionality.

- These helpers support:

- hydrofabric inspection

- HRU-to-segment connectivity checks

- parameter mapping and visualization

- streamflow observation handling

- output visualization for variables like recharge or evapotranspiration

- The implementation relies heavily on geospatial data handling plus plotting/mapping libraries.

7. Visualization output subsystem

- Interactive plots are built with plotly, and maps are built with folium.

- Generated artifacts are written to notebook output folders as HTML, so they are decoupled from Jupyter and can be opened in a browser or embedded elsewhere.

- This output boundary is important: notebooks produce portable HTML deliverables, not just inline notebook cells.

8. Calibration subsystem

- pestpp_ies_calibration is a distinct calibration workspace, not just a helper module.

- It includes:

- calibration templates and ancillary CSVs

- forward-run and post-processing scripts

- hotstart/result transfer utilities

- Condor-related preparation scripts for distributed execution

- The calibration flow is centered on preparing PEST/PEST++ IES inputs, executing model runs, and consolidating results.

9. Vendored dependency boundary

- The PEST workflow vendors a local copy of pyemu under pestpp_ies_calibration/dependencies/pyemu.

- This suggests the calibration workflow is designed to be self-contained and resilient to upstream dependency changes.

- In contrast, the rest of the repo uses the shared environment in environment.yaml.

10. Domain data and static inputs

- domain_data/ holds example or user-provided NHM subdomains, including a concrete Walla_Walla example.

- This area contains model control files, NetCDF outputs, geopackages, parameter files, and observation data.

- data_dependencies/ stores reusable static inputs such as shapefiles, CSV lookup tables, images, and explanation tables used by notebooks.

11. Data flow through the system

- External or provided model data is pulled into domain_data/ using pull_domain.py.

- Workflow notebooks read from domain_data/ and data_dependencies/, then call helper modules from src/assist or legacy mirrors.

- Outputs are written back into domain-specific notebook output folders as plots, maps, and calibration artifacts.

12. Legacy/backward-compatibility layout

- There are top-level copies of older helper packages such as nhm_helpers/, nhf_assist/, and pestpp_ies_calibration/helpers/.

- These appear to support older notebook paths and import conventions alongside the newer src/assist package layout.

- For architecture diagrams, treat src/assist as the current library core and the top-level helper directories as compatibility mirrors or workflow-local support code.

13. Automation and testing

- GitHub Actions CI lives under .github/workflows, with helper scripts for notebook testing and cache management.

- The CI setup indicates notebooks are part of the test surface, so repository validation likely includes generating notebooks and running selected notebook checks.

- Pre-commit and packaging metadata (pyproject.toml, environment.yaml) reinforce that this is a reproducible Python workflow repository, not just a notebook dump.

14. Main external/runtime dependencies

- The runtime is Python in a conda/mamba environment.

- Key external libraries shaping the architecture are pywatershed for NHM simulation, plotly and folium for visualization, pyemu/PEST++ for calibration, and geospatial tooling for shapefile/geopackage handling.

- The repository also references HyTEST OSN storage as the source for downloadable example domains.