# ajkhattak/basin_workflow architecture

> Architecture diagram of the GitHub repository ajkhattak/basin_workflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ajkhattak/basin_workflow
- Repository: https://github.com/ajkhattak/basin_workflow

## Overview

##### Project type and purpose
- This is a domain‐specific workflow tool (“basin_workflow”) for hydrological modeling, data preparation, calibration, and simulation.
- It is neither a web app nor a microservices suite; it’s a CLI‐driven, multi‐language (Python + R + external binaries) pipeline that:
- Loads configuration (YAML)
- Downloads or preprocesses forcing data
- Generates model input files
- Runs hydrologic models (R scripts, external Fortran/C tools via Python)
- Calibrates with plugin framework (ngen_cal_conf + ngen_cal_plugins)
- Produces outputs and stores results (local FS, S3)

##### File‐structure analysis
- Top‐level “basin_workflow” directory contains:
- configs/ (YAML templates for workflow + calibration)
- extern/ (third‐party calibration frameworks + plugins)
- generate_files/ (Python modules to build all input files)
- giuh_twi/ (R scripts implementing GIUH/TWI models)
- utils/ (shell + Python + R utilities: S3 download, NWM ingestion, EC2 setup)
- main.py & runner.py (entry-point CLI & orchestration)
- Other artifacts: packaging configs (.gitmodules, setup.cfg), documentation, installation scripts.

##### README insights
- Generic template but points to INSTALL.md, CONTRIBUTING.md, etc.

a. Components to include
- CLI Layer: basin_workflow/main.py
- Orchestrator / Workflow Runner: runner.py
- Configuration Store: configs/*.yaml
- Data Ingestion Modules:
– Python forcing downloader (utils/python/download_nwm_streamflow.py)
– R S3 downloader (utils/R/download_from_s3_bucket.R)
- Input‐file Generators: generate_files/*.py
- Model Execution Engines:
– R GIUH/TWI scripts (giuh_twi/*.R)
– External calibration binaries (extern/CIROH_DL_NextGen, ngen-cal)
- Plugin Framework: extern/ngen_cal_conf & extern/ngen_cal_plugins
- Output & Storage: local output directories, AWS S3 buckets
- Infrastructure Setup: utils/setup_ec2.sh, utils/build_models.sh

b. Relationships & interactions
1. User invokes CLI → CLI loads configuration
2. Orchestrator reads YAML → triggers data ingestion modules
3. Ingested data stored locally or in S3
4. Input‐file generators consume data + config → produce model inputs
5. Orchestrator dispatches:
– R scripts (spawn R process)
– ngen-cal tool for calibration (via Python wrappers)
6. Plugin hooks execute (save iteration output, sim vs obs)
7. Results aggregated → written to local FS or pushed back to S3

c. Architectural patterns & principles
- Pipeline (DAG) architecture: tasks executed in sequence or parallel, per config
- Modular plugin pattern: external calibration plugins loaded at runtime
- Polyglot integration: orchestrator bridges Python, R, and compiled binaries
- Configuration‐driven: YAML files steer the entire workflow

d. Key technologies & frameworks
- Python 3 (CLI, orchestrator, I/O modules)
- R (giuh_twi modeling)
- AWS S3 (via R + Python utils)
- ngen‐cal (external calibration tool)
- YAML (configuration)

##### Final advice
- Be exhaustive in capturing each folder as a logical component.
- Don’t over‐simplify: show the multi‐language orchestration.
- Use a pipeline layout (left‐to‐right) reflecting the order of operations.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layers and components
    CLI["CLI Layer\nmain.py"]:::pythonProc
    Orchestrator["Orchestrator\nrunner.py"]:::pythonProc
    subgraph "Configuration Store"
        direction TB
        CFG1["config_workflow.yaml"]:::config
        CFG2["config_calib.yaml"]:::config
        CFG3["calib_params.yaml"]:::config
        CFG4["config_aorc.yaml"]:::config
        CFG5["config_troute.yaml"]:::config
    end
    subgraph "Data Ingestion Modules"
        direction TB
        IngestPy["download_nwm_streamflow.py"]:::pythonProc
        IngestRS3["download_from_s3_bucket.R"]:::rProc
        IngestRNWM["nwm_data.R"]:::rProc
    end
    subgraph "Input-file Generators"
        direction TB
        Generators["generate_files/*.py"]:::pythonProc
    end
    subgraph "Model Execution Engines"
        direction TB
        GIUH_TWI["giuh_twi/*.R"]:::rProc
        ExtBin1["CIROH_DL_NextGen"]:::ext
        ExtBin2["ngen-cal"]:::ext
    end
    subgraph "Plugin Framework"
        direction TB
        PluginCore["ngen_cal_conf"]:::ext
        PluginSrc["ngen_cal_plugins"]:::ext
    end
    subgraph "Output & Storage"
        direction TB
        LocalFS["Local File System"]:::store
        S3["AWS S3 Bucket"]:::store
    end
    subgraph "Infrastructure Setup"
        direction TB
        EC2["setup_ec2.sh"]:::infra
        BuildModels["build_models.sh"]:::infra
    end

    %% Data flow
    CLI -->|"invokes"| Orchestrator
    Orchestrator -->|"loads"| CFG1
    Orchestrator --> CFG2
    Orchestrator --> CFG3
    Orchestrator --> CFG4
    Orchestrator --> CFG5
    Orchestrator -->|"triggers"| IngestPy
    Orchestrator --> IngestRS3
    Orchestrator --> IngestRNWM
    IngestPy -->|"stores data"| LocalFS
    IngestRS3 --> LocalFS
    IngestRNWM --> LocalFS
    LocalFS -->|"feeds"| Generators
    CFG1 --> Generators
    Generators -->|"produces"| GIUH_TWI
    Generators --> ExtBin1
    Generators --> ExtBin2
    GIUH_TWI --> PluginCore
    ExtBin1 --> PluginCore
    ExtBin2 --> PluginCore
    PluginCore --> PluginSrc
    PluginSrc -->|"writes results"| LocalFS
    PluginSrc -->|"uploads"| S3
    %% Infrastructure hooks
    Orchestrator -.-> EC2
    Orchestrator -.-> BuildModels

    %% Click Events
    click CLI "https://github.com/ajkhattak/basin_workflow/blob/master/basin_workflow/main.py"
    click Orchestrator "https://github.com/ajkhattak/basin_workflow/blob/master/basin_workflow/runner.py"
    click CFG1 "https://github.com/ajkhattak/basin_workflow/blob/master/basin_workflow/configs/config_workflow.yaml"
    click CFG2 "https://github.com/ajkhattak/basin_workflow/blob/master/basin_workflow/configs/config_calib.yaml"
    click CFG3 "https://github.com/ajkhattak/basin_workflow/blob/master/basin_workflow/configs/calib_params.yaml"
    click CFG4 "https://github.com/ajkhattak/basin_workflow/blob/master/basin_workflow/configs/config_aorc.yaml"
    click CFG5 "https://github.com/ajkhattak/basin_workflow/blob/master/basin_workflow/configs/samples/config_troute.yaml"
    click IngestPy "https://github.com/ajkhattak/basin_workflow/blob/master/basin_workflow/utils/python/download_nwm_streamflow.py"
    click IngestRS3 "https://github.com/ajkhattak/basin_workflow/blob/master/basin_workflow/utils/R/download_from_s3_bucket.R"
    click IngestRNWM "https://github.com/ajkhattak/basin_workflow/blob/master/basin_workflow/utils/R/nwm_data.R"
    click Generators "https://github.com/ajkhattak/basin_workflow/tree/master/basin_workflow/generate_files/"
    click GIUH_TWI "https://github.com/ajkhattak/basin_workflow/tree/master/basin_workflow/giuh_twi/"
    click ExtBin1 "https://github.com/ajkhattak/basin_workflow/tree/master/basin_workflow/extern/CIROH_DL_NextGen"
    click ExtBin2 "https://github.com/ajkhattak/basin_workflow/tree/master/basin_workflow/extern/ngen-cal"
    click PluginCore "https://github.com/ajkhattak/basin_workflow/tree/master/basin_workflow/extern/ngen_cal_conf"
    click PluginSrc "https://github.com/ajkhattak/basin_workflow/tree/master/basin_workflow/extern/ngen_cal_plugins/src/"
    click EC2 "https://github.com/ajkhattak/basin_workflow/blob/master/basin_workflow/utils/setup_ec2.sh"
    click BuildModels "https://github.com/ajkhattak/basin_workflow/blob/master/basin_workflow/utils/build_models.sh"

    %% Styles
    classDef pythonProc fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef rProc fill:#90EE90,stroke:#333,stroke-width:1px
    classDef ext fill:#FFA500,stroke:#333,stroke-width:1px
    classDef store fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef config fill:#E6E6FA,stroke:#333,stroke-width:1px
    classDef infra fill:#F0E68C,stroke:#333,stroke-width:1px
```
