# ciroh-ua/ngen-datastream architecture

> Architecture diagram of the GitHub repository ciroh-ua/ngen-datastream, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ciroh-ua/ngen-datastream
- Repository: https://github.com/ciroh-ua/ngen-datastream

## Overview

##### Main Components

a) Core System Components:
- Data Collection Service
- Forcing Processor
- NextGen Configuration Generator
- NextGen Execution Orchestrator (NGIAB)
- Output Handler

b) External Components:
- NextGen Water Modeling Framework (as an external dependency)
- National Water Model (data source)
- Hydrofabric Data Source

##### Specific Implementation Details

Create distinct sections for:

a) Input Processing:
- Hydrofabric file handling
- Forcing data collection
- Configuration generation

b) Execution Environment:
- Docker containerization (`docker/` directory)
- Multi-process execution support

c) AWS Integration:
- Lambda functions (from terraform/lambda_functions/)
- Infrastructure automation

Create clear boundaries between:
- Local Execution Environment
- Cloud Infrastructure
- Data Processing Pipeline
- Model Execution Layer

Key Technical Elements to Highlight:
- BMI Configuration Generation
- Multi-model Support
- Parallel Processing
- Infrastructure as Code (Terraform)
- Container Orchestration
- Lambda Function Integration

Remember to include version information and data flow direction indicators, as this system integrates with various external components and data sources.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph User_Layer
        CLI["CLI Interface"]:::cli
        Config["Configuration Management"]:::config
    end

    subgraph Processing_Layer
        DataCollect["Data Collection Service"]:::core
        ForcingProc["Forcing Processor"]:::core
        ConfigGen["Configuration Generator"]:::core
        NGConfigGen["NextGen Configuration Generator"]:::core
    end

    subgraph Execution_Layer
        ExecOrch["Execution Orchestrator (NGIAB)"]:::exec
        ModelConfig["Model Configurations"]:::config
        OutputHandler["Output Handler"]:::core
    end

    subgraph Infrastructure_Layer
        Docker["Docker Infrastructure"]:::infra
        AWS["AWS Infrastructure"]:::infra
        Lambda["Lambda Functions"]:::infra
    end

    subgraph External_Systems
        NextGen["NextGen Water Modeling Framework"]:::external
        NWM["National Water Model"]:::external
        Hydrofabric["Hydrofabric Data Source"]:::external
    end

    %% Data Flow Relationships
    NWM -->|Input Data| DataCollect
    Hydrofabric -->|Input Data| ConfigGen
    DataCollect -->|Forcing Data| ForcingProc
    ForcingProc -->|Processed Data| NGConfigGen
    ConfigGen -->|System Config| NGConfigGen
    NGConfigGen -->|Model Setup| ExecOrch
    ModelConfig -->|Parameters| ExecOrch
    ExecOrch -->|Results| OutputHandler

    %% Control Flow
    CLI -->|Commands| Config
    Config -->|Settings| ConfigGen
    AWS -->|Manages| Lambda
    Docker -->|Containers| ExecOrch
    ExecOrch -->|Executes| NextGen

    %% Style Definitions
    classDef core fill:#3498db,stroke:#2980b9,color:white
    classDef config fill:#e67e22,stroke:#d35400,color:white
    classDef exec fill:#2ecc71,stroke:#27ae60,color:white
    classDef infra fill:#9b59b6,stroke:#8e44ad,color:white
    classDef external fill:#95a5a6,stroke:#7f8c8d,color:white
    classDef cli fill:#e74c3c,stroke:#c0392b,color:white

    %% Click Events
    click CLI "https://github.com/CIROH-UA/ngen-datastream/blob/main/scripts/stream.sh"
    click Config "https://github.com/CIROH-UA/ngen-datastream/tree/main/configs/ngen/"
    click DataCollect "https://github.com/CIROH-UA/ngen-datastream/blob/main/forcingprocessor/src/forcingprocessor/nwm_filenames_generator.py"
    click ForcingProc "https://github.com/CIROH-UA/ngen-datastream/blob/main/forcingprocessor/src/forcingprocessor/processor.py"
    click ConfigGen "https://github.com/CIROH-UA/ngen-datastream/blob/main/python_tools/src/python_tools/configure_datastream.py"
    click NGConfigGen "https://github.com/CIROH-UA/ngen-datastream/blob/main/python_tools/src/python_tools/ngen_configs_gen.py"
    click ExecOrch "https://github.com/CIROH-UA/ngen-datastream/blob/main/research_datastream/scripts/ngen_datastream_parallel.sh"
    click ModelConfig "https://github.com/CIROH-UA/ngen-datastream/blob/main/configs/ngen/realization_sloth_nom_cfe_pet.json"
    click OutputHandler "https://github.com/CIROH-UA/ngen-datastream/blob/main/python_tools/src/python_tools/plot_datastream.py"
    click Docker "https://github.com/CIROH-UA/ngen-datastream/tree/main/docker/"
    click AWS "https://github.com/CIROH-UA/ngen-datastream/tree/main/research_datastream/terraform/"
    click Lambda "https://github.com/CIROH-UA/ngen-datastream/tree/main/research_datastream/terraform/lambda_functions/"
```
