How noaa-owp/formulation-selector works
High-level components
• Orchestrator/CLI layer
• Scripts under /scripts drive end-to-end workflows: data ingest, attribute grabbing, training, evaluation, prediction, visualization.
• They consume YAML configuration files and invoke package APIs.
• Attribute‐grabber (R)
• R package proc.attr.hydfab under pkg/proc.attr.hydfab
• Fetches catchment attributes via NOAA-OWP/hydrofabric and USGS NHDPlus, writes out Parquet files.
• Data‐preprocessing (Python)
• Python package fs_prep under pkg/fs_prep
• Standardizes raw model performance metrics into a common schema.
• Feature‐selection & ML (Python)
• Python package fs_algo under pkg/fs_algo
• Implements training (fs_algo_train_eval.py), performance visualization (fs_perf_viz.py), prediction (fs_pred_algo.py), preprocessing (fs_proc_algo.py), transformation of attributes (fs_tfrm_attrs*.py).
• External services & data
• NOAA-OWP/hydrofabric REST or file store
• USGS nhdplusTools
• User’s raw metrics datasets, YAML configs, CSV/Parquet/NetCDF files
• Third-party Python libs (pandas, xarray, scikit-learn, dask)
Architectural patterns & design principles
• Separation of concerns: R handles data acquisition; Python handles preprocessing and ML.
• Config-driven: YAML files decouple scripts from hard-coded parameters.
• Modular packages: fs_prep and fs_algo expose clear APIs; tests under tests/.
Technologies & frameworks
• Python: fs_prep, fs_algo (pandas, xarray, scikit-learn, joblib, dask)
• Build & docs: Sphinx (docs/), Makefiles, GitHub Actions.
Detail is essential
• Major script folder (e.g., /scripts/eval_ingest, /scripts/prediction) and map to the packages they invoke.
• File I/O: e.g., scripts/eval_ingest/xssa/proc_xssa_metrics.py reads xssa_schema.yaml → calls fs_prep.proc_col_schema() → writes *.parquet.