How ajkhattak/basin_workflow works
Project type and purpose
• This is a domain‐specific workflow tool (“basin_workflow”) for hydrological modeling, data preparation, calibration, and simulation.
• It is neither a web app nor a microservices suite; it’s a CLI‐driven, multi‐language (Python + R + external binaries) pipeline that:
• Loads configuration (YAML)
• Downloads or preprocesses forcing data
• Generates model input files
• Runs hydrologic models (R scripts, external Fortran/C tools via Python)
• Calibrates with plugin framework (ngen_cal_conf + ngen_cal_plugins)
• Produces outputs and stores results (local FS, S3)
File‐structure analysis
• Top‐level “basin_workflow” directory contains:
• configs/ (YAML templates for workflow + calibration)
• extern/ (third‐party calibration frameworks + plugins)
• generate_files/ (Python modules to build all input files)
• giuh_twi/ (R scripts implementing GIUH/TWI models)
• utils/ (shell + Python + R utilities: S3 download, NWM ingestion, EC2 setup)
• main.py & runner.py (entry-point CLI & orchestration)
• Other artifacts: packaging configs (.gitmodules, setup.cfg), documentation, installation scripts.
README insights
• Generic template but points to INSTALL.md, CONTRIBUTING.md, etc.
a. Components to include
• CLI Layer: basin_workflow/main.py
• Orchestrator / Workflow Runner: runner.py
• Configuration Store: configs/*.yaml
• Data Ingestion Modules:
– Python forcing downloader (utils/python/download_nwm_streamflow.py)
– R S3 downloader (utils/R/download_from_s3_bucket.R)
• Input‐file Generators: generate_files/*.py
• Model Execution Engines:
– R GIUH/TWI scripts (giuh_twi/*.R)
– External calibration binaries (extern/CIROH_DL_NextGen, ngen-cal)
• Plugin Framework: extern/ngen_cal_conf & extern/ngen_cal_plugins
• Output & Storage: local output directories, AWS S3 buckets
• Infrastructure Setup: utils/setup_ec2.sh, utils/build_models.sh
b. Relationships & interactions
1. User invokes CLI → CLI loads configuration
2. Orchestrator reads YAML → triggers data ingestion modules
3. Ingested data stored locally or in S3
4. Input‐file generators consume data + config → produce model inputs
5. Orchestrator dispatches:
– R scripts (spawn R process)
– ngen-cal tool for calibration (via Python wrappers)
6. Plugin hooks execute (save iteration output, sim vs obs)
7. Results aggregated → written to local FS or pushed back to S3
c. Architectural patterns & principles
• Pipeline (DAG) architecture: tasks executed in sequence or parallel, per config
• Modular plugin pattern: external calibration plugins loaded at runtime
• Polyglot integration: orchestrator bridges Python, R, and compiled binaries
• Configuration‐driven: YAML files steer the entire workflow
d. Key technologies & frameworks
• Python 3 (CLI, orchestrator, I/O modules)
• R (giuh_twi modeling)
• AWS S3 (via R + Python utils)
• ngen‐cal (external calibration tool)
• YAML (configuration)
Final advice
• Be exhaustive in capturing each folder as a logical component.
• Don’t over‐simplify: show the multi‐language orchestration.
• Use a pipeline layout (left‐to‐right) reflecting the order of operations.