# noaa-owp/formulation-selector architecture

> Architecture diagram of the GitHub repository noaa-owp/formulation-selector, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/noaa-owp/formulation-selector
- Repository: https://github.com/noaa-owp/formulation-selector

## Overview

##### High-level components
- **Orchestrator/CLI layer**
- Scripts under `/scripts` drive end-to-end workflows: data ingest, attribute grabbing, training, evaluation, prediction, visualization.
- They consume YAML configuration files and invoke package APIs.
- **Attribute‐grabber (R)**
- R package `proc.attr.hydfab` under `pkg/proc.attr.hydfab`
- Fetches catchment attributes via NOAA-OWP/hydrofabric and USGS NHDPlus, writes out Parquet files.
- **Data‐preprocessing (Python)**
- Python package `fs_prep` under `pkg/fs_prep`
- Standardizes raw model performance metrics into a common schema.
- **Feature‐selection & ML (Python)**
- Python package `fs_algo` under `pkg/fs_algo`
- Implements training (`fs_algo_train_eval.py`), performance visualization (`fs_perf_viz.py`), prediction (`fs_pred_algo.py`), preprocessing (`fs_proc_algo.py`), transformation of attributes (`fs_tfrm_attrs*.py`).
- **External services & data**
- NOAA-OWP/hydrofabric REST or file store
- USGS nhdplusTools
- User’s raw metrics datasets, YAML configs, CSV/Parquet/NetCDF files
- Third-party Python libs (pandas, xarray, scikit-learn, dask)

##### Architectural patterns & design principles
- **Separation of concerns**: R handles data acquisition; Python handles preprocessing and ML.
- **Config-driven**: YAML files decouple scripts from hard-coded parameters.
- **Modular packages**: `fs_prep` and `fs_algo` expose clear APIs; tests under `tests/`.

##### Technologies & frameworks
- Python: `fs_prep`, `fs_algo` (pandas, xarray, scikit-learn, joblib, dask)
- Build & docs: Sphinx (docs/), Makefiles, GitHub Actions.

##### Detail is essential
- Major script folder (e.g., `/scripts/eval_ingest`, `/scripts/prediction`) and map to the packages they invoke.
- File I/O: e.g., `scripts/eval_ingest/xssa/proc_xssa_metrics.py` reads `xssa_schema.yaml` → calls `fs_prep.proc_col_schema()` → writes `*.parquet`.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestrator/CLI Layer
    subgraph "Orchestrator/CLI Layer"
        direction TB
        CLI_SCRIPTS["scripts/ (CLI Entry)"]:::orchestrator
        EVAL_INGEST["scripts/eval_ingest/"]:::orchestrator
        PREDICTION["scripts/prediction/"]:::orchestrator
        ANALYSIS["scripts/analysis/"]:::orchestrator
        CONFIG_DIR["scripts/config/"]:::orchestrator
    end

    %% Configuration
    YAML_CONFIG["YAML Config"]:::external

    %% R Attribute Grabber
    subgraph "R: Attribute Grabber" 
        direction TB
        R_PKG["pkg/proc.attr.hydfab/"]:::rComp
        R_FUNC1["proc_attr_grabber.R"]:::rComp
        R_FUNC2["rafts_utils.R"]:::rComp
        R_FLOW1["fs_attrs_grab.R"]:::rComp
        R_FLOW2["fs_attrs_miss.R"]:::rComp
        R_DATA["inst/extdata/ (Parquet Outputs)"]:::external
    end

    %% Python Data Preprocessing
    subgraph "Python: fs_prep"
        direction TB
        FS_PREP_PKG["pkg/fs_prep/fs_prep/"]:::pythonComp
        FS_PREP_CORE["proc_eval_metrics.py"]:::pythonComp
        FS_PREP_SCHEMA["data/fs_categories.yaml"]:::pythonComp
        FS_PREP_TESTS["tests/"]:::pythonComp
        FS_PREP_OUTPUT["Standardized Metrics (Parquet)"]:::external
    end

    %% Python Feature Selection & ML
    subgraph "Python: fs_algo"
        direction TB
        FS_ALGO_PKG["pkg/fs_algo/fs_algo/"]:::pythonComp
        FS_TRAIN["fs_algo_train_eval.py"]:::pythonComp
        FS_VIZ["fs_perf_viz.py"]:::pythonComp
        FS_PRED["fs_pred_algo.py"]:::pythonComp
        FS_PROC["fs_proc_algo.py"]:::pythonComp
        FS_PROC_VIZ["fs_proc_algo_viz.py"]:::pythonComp
        FS_TFRM["fs_tfrm_attrs.py"]:::pythonComp
        FS_TFRM_SINGLE["fs_tfrm_attrs_single_locs.py"]:::pythonComp
        FS_ALGO_DOCS["docs/"]:::pythonComp
        FS_ALGO_SETUP["setup.py"]:::pythonComp
        MODEL_OUTPUT["Model Artifacts & Eval Metrics"]:::external
        PRED_OUTPUT["Predictions"]:::external
        PLOTS_OUTPUT["Performance Plots (PNG/SVG)"]:::external
    end

    %% External Services & Data
    subgraph "External Services & Data"
        direction TB
        NOAA["NOAA-OWP/hydrofabric"]:::external
        USGS["USGS NHDPlus"]:::external
        RAW_METRICS["Raw Metrics (CSV/NetCDF)"]:::external
        THIRD_PARTY_LIBS["Third-party Python Libs"]:::external
    end

    %% CI & Documentation
    subgraph "CI & Documentation"
        direction TB
        GH_ACTIONS[".github/workflows/add_to_project.yml"]:::external
        SPHINX_DOCS["Sphinx docs/"]:::external
    end

    %% Connections
    CLI_SCRIPTS -->|invokes| EVAL_INGEST
    CLI_SCRIPTS -->|invokes| PREDICTION
    CLI_SCRIPTS -->|invokes| ANALYSIS
    CLI_SCRIPTS -->|reads| CONFIG_DIR
    CONFIG_DIR -->|provides| YAML_CONFIG

    EVAL_INGEST -->|uses config| YAML_CONFIG
    EVAL_INGEST -->|calls| R_PKG
    R_PKG -->|executes| R_FLOW1
    R_PKG -->|executes| R_FLOW2
    R_PKG -->|uses| R_FUNC1
    R_PKG -->|uses| R_FUNC2
    R_PKG -->|fetches from| NOAA
    R_PKG -->|fetches from| USGS
    R_PKG -->|writes| R_DATA

    EVAL_INGEST -->|calls| FS_PREP_CORE
    FS_PREP_CORE -->|reads| RAW_METRICS
    FS_PREP_CORE -->|uses| FS_PREP_SCHEMA
    FS_PREP_CORE -->|writes| FS_PREP_OUTPUT
    FS_PREP_CORE -->|tested by| FS_PREP_TESTS

    CLI_SCRIPTS -->|calls| FS_TRAIN
    FS_TRAIN -->|reads| FS_PREP_OUTPUT
    FS_TRAIN -->|reads| R_DATA
    FS_TRAIN -->|writes| MODEL_OUTPUT

    CLI_SCRIPTS -->|calls| FS_PRED
    FS_PRED -->|reads| R_DATA
    FS_PRED -->|reads| MODEL_OUTPUT
    FS_PRED -->|writes| PRED_OUTPUT

    CLI_SCRIPTS -->|calls| FS_VIZ
    FS_VIZ -->|reads| MODEL_OUTPUT
    FS_VIZ -->|writes| PLOTS_OUTPUT

    FS_PROC -->|prep for train| FS_TRAIN
    FS_PROC_VIZ -->|prep visualization| FS_VIZ
    FS_TFRM -->|transform attrs| FS_TRAIN
    FS_TFRM_SINGLE -->|transform attrs| FS_PRED

    GH_ACTIONS -->|CI for| FS_PREP_PKG
    GH_ACTIONS -->|CI for| FS_ALGO_PKG
    SPHINX_DOCS -->|docs for| FS_PREP_PKG
    SPHINX_DOCS -->|docs for| FS_ALGO_PKG

    %% Click Events
    click CLI_SCRIPTS "https://github.com/noaa-owp/formulation-selector/tree/main/scripts/"
    click EVAL_INGEST "https://github.com/noaa-owp/formulation-selector/tree/main/scripts/eval_ingest/"
    click PREDICTION "https://github.com/noaa-owp/formulation-selector/tree/main/scripts/prediction/"
    click ANALYSIS "https://github.com/noaa-owp/formulation-selector/tree/main/scripts/analysis/"
    click CONFIG_DIR "https://github.com/noaa-owp/formulation-selector/tree/main/scripts/config/"

    click R_PKG "https://github.com/noaa-owp/formulation-selector/tree/main/pkg/proc.attr.hydfab/"
    click R_FUNC1 "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/proc.attr.hydfab/R/proc_attr_grabber.R"
    click R_FUNC2 "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/proc.attr.hydfab/R/rafts_utils.R"
    click R_FLOW1 "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/proc.attr.hydfab/flow/fs_attrs_grab.R"
    click R_FLOW2 "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/proc.attr.hydfab/flow/fs_attrs_miss.R"
    click R_DATA "https://github.com/noaa-owp/formulation-selector/tree/main/pkg/proc.attr.hydfab/inst/extdata/"

    click FS_PREP_PKG "https://github.com/noaa-owp/formulation-selector/tree/main/pkg/fs_prep/fs_prep/"
    click FS_PREP_CORE "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/fs_prep/fs_prep/proc_eval_metrics.py"
    click FS_PREP_SCHEMA "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/fs_prep/fs_prep/data/fs_categories.yaml"
    click FS_PREP_TESTS "https://github.com/noaa-owp/formulation-selector/tree/main/pkg/fs_prep/fs_prep/tests/"

    click FS_ALGO_PKG "https://github.com/noaa-owp/formulation-selector/tree/main/pkg/fs_algo/fs_algo/"
    click FS_TRAIN "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/fs_algo/fs_algo/fs_algo_train_eval.py"
    click FS_VIZ "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/fs_algo/fs_algo/fs_perf_viz.py"
    click FS_PRED "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/fs_algo/fs_algo/fs_pred_algo.py"
    click FS_PROC "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/fs_algo/fs_algo/fs_proc_algo.py"
    click FS_PROC_VIZ "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/fs_algo/fs_algo/fs_proc_algo_viz.py"
    click FS_TFRM "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/fs_algo/fs_algo/fs_tfrm_attrs.py"
    click FS_TFRM_SINGLE "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/fs_algo/fs_algo/fs_tfrm_attrs_single_locs.py"
    click FS_ALGO_DOCS "https://github.com/noaa-owp/formulation-selector/tree/main/pkg/fs_algo/docs/"
    click FS_ALGO_SETUP "https://github.com/noaa-owp/formulation-selector/blob/main/pkg/fs_algo/setup.py"

    classDef rComp fill:#ADD8E6,stroke:#000
    classDef pythonComp fill:#90EE90,stroke:#000
    classDef external fill:#D3D3D3,stroke:#000
    classDef orchestrator fill:#FFFACD,stroke:#000
```
