# red-hat-ai-innovation-team/sdg_hub architecture

> Architecture diagram of the GitHub repository red-hat-ai-innovation-team/sdg_hub, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/red-hat-ai-innovation-team/sdg_hub
- Repository: https://github.com/red-hat-ai-innovation-team/sdg_hub

## Overview

##### Project type and purpose
- **Type:** A Python‐based SDK / toolkit for building “no‐code” synthetic data generation pipelines for LLMs, plus a minimal web UI.

##### File structure
Top‐level directories:
- **src/sdg_hub/** – Core library
– `blocks/` (Block implementations: LLMBlock, FilterByValueBlock, util blocks, etc.)
– `configs/` (YAML prompt templates and block settings)
– `flows/` (Sample flow definitions)
– `flow.py`, `flow_runner.py`, `pipeline.py`, `registry.py`, `checkpointer.py`
– `prompts.py`, `sdg.py` (CLI entrypoint), `logger_config.py`
- **examples/** – Jupyter notebooks and sample flows demonstrating usage
- **web_interface/** – A Flask (or lightweight) web app (`app.py`) serving a JS/HTML front end to run flows interactively
- **tests/** – Unit tests for blocks, flows, runner
- **scripts/** – Dev helpers (e.g., lint wrappers)
Configuration and CI: `pyproject.toml`, GitHub Actions workflows, pre-commit, linters.

##### Insights from the README
- **Core concepts:** Blocks, Flows, Prompts (YAML), Checkpoints, HuggingFace Datasets.
- **Design principles:** Modular, no-code, scalable, checkpoint-resumable.

a. **Main components**
1. **User Interface Layer**
– CLI (`sdg.py`)
– Web UI (`web_interface/app.py` + JS/CSS/HTML)
2. **Orchestration Layer**
– **Flow Loader**: Reads YAML flow file
– **FlowRunner** + **Registry**: Maps `block_type` → Block class
– **Pipeline**: Constructs ordered graph of Blocks
3. **Execution Layer**
– **Blocks**: Instances of LLMBlock, FilterByValueBlock, IterBlock, utility blocks
– **Config Loader**: Loads block/prompt YAML under `configs/`
– **Checkpointer**: Persists intermediate HuggingFace datasets to disk
4. **Data Layer**
5. **External Services**
– **LLM API endpoints** (HuggingFace Hub, OpenAI, or other model endpoints)
b. **Relationships & Interactions**
1. UI triggers the **FlowRunner**, passing the path to a YAML flow.
2. **FlowRunner** uses **Registry** to instantiate Block objects in order.
3. **Pipeline** executes each Block in sequence or nested iterations:
– Blocks load their **prompt/config YAML** (via `configs/`)
– **LLMBlock** issues HTTP calls to an LLM service and awaits a response
– Other blocks transform/filter the dataset in memory
4. Final dataset is returned to the CLI (printed or written) or sent back to the Web UI for display.
c. **Architectural patterns & principles**
– **Registry/Factory pattern** for block instantiation
– **Pipeline pattern** for chaining computational steps
– **Config‐driven design** (no code changes needed to build new flows)
– **Modularity & reusability**: blocks are self-contained, interchangeable units
– **Checkpointing** for fault tolerance and resumability
d. **Key technologies & libraries**
– HTTP clients for LLM APIs (requests or huggingface-hub)
– Flask (or lightweight WSGI) for `web_interface/app.py`
– JavaScript/CSS/HTML for the web front end

##### Detail
- Individual modules (e.g., `flow_runner.py`, `registry.py`, `blocks/llmblock.py`) only if necessary for clarity.
- Essential layers: Config → Engine → Data → External.
- Do not over-abstract: include the Checkpointer and HuggingFace Datasets as first-class components.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface Layer
    subgraph "User Interface Layer"
        CLI["CLI: sdg"]:::core
        WebUI["Flask Web UI"]:::core
        subgraph "Web UI Assets"
            CSS["CSS: style.css"]:::config
            JS["JS: app.js"]:::config
            HTML["HTML: index.html"]:::config
        end
    end

    %% Config Layer
    subgraph "Config Layer"
        ConfigFiles["YAML Configs"]:::config
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        FlowLoader["Flow Loader\n(flow.py)"]:::core
        FlowRunner["FlowRunner\n(flow_runner.py)"]:::core
        Registry["Registry\n(registry.py)"]:::core
        Pipeline["Pipeline\n(pipeline.py)"]:::core
    end

    %% Execution Layer
    subgraph "Execution Layer"
        subgraph "Block Implementations"
            BlockBase["block.py"]:::core
            LLMBlock["llmblock.py"]:::core
            UtilBlocks["utilblocks.py"]:::core
        end
        ConfigLoader["Config Loader"]:::config
        Prompts["Prompts\n(prompts.py)"]:::core
        Checkpointer["Checkpointer\n(checkpointer.py)"]:::core
    end

    %% Data Layer
    subgraph "Data Layer"
        Dataset["HuggingFace Datasets\n(datautils.py)"]:::data
        Storage[(Checkpoint Storage\nArrow files)]:::data
    end

    %% External Services
    External["LLM API"]:::external

    %% Connections
    CLI -->|"invoke"| FlowRunner
    WebUI -->|"invoke"| FlowRunner
    CSS --> WebUI
    JS --> WebUI
    HTML --> WebUI

    FlowRunner -->|"loads flow"| FlowLoader
    ConfigFiles --> FlowLoader
    FlowLoader --> Registry
    Registry --> Pipeline

    Pipeline --> BlockBase
    Pipeline --> LLMBlock
    Pipeline --> UtilBlocks

    ConfigFiles --> ConfigLoader
    ConfigLoader --> BlockBase
    ConfigLoader --> LLMBlock
    ConfigLoader --> UtilBlocks

    Prompts --> BlockBase
    Prompts --> LLMBlock
    Prompts --> UtilBlocks

    BlockBase --> Dataset
    LLMBlock -->|HTTP request/response| External
    LLMBlock --> Dataset
    UtilBlocks --> Dataset

    Dataset --> Pipeline
    Pipeline --> Dataset

    LLMBlock --> Checkpointer
    UtilBlocks --> Checkpointer
    Checkpointer --> Storage

    Dataset -->|"output dataset"| CLI
    Dataset -->|"output dataset"| WebUI

    %% Click Events
    click CLI "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/src/sdg_hub/sdg.py"
    click WebUI "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/web_interface/app.py"
    click CSS "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/web_interface/static/css/style.css"
    click JS "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/web_interface/static/js/app.js"
    click HTML "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/web_interface/templates/index.html"
    click FlowLoader "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/src/sdg_hub/flow.py"
    click FlowRunner "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/src/sdg_hub/flow_runner.py"
    click Registry "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/src/sdg_hub/registry.py"
    click Pipeline "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/src/sdg_hub/pipeline.py"
    click BlockBase "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/src/sdg_hub/blocks/block.py"
    click LLMBlock "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/src/sdg_hub/blocks/llmblock.py"
    click UtilBlocks "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/src/sdg_hub/blocks/utilblocks.py"
    click ConfigFiles "https://github.com/red-hat-ai-innovation-team/sdg_hub/tree/main/src/sdg_hub/configs/"
    click ConfigLoader "https://github.com/red-hat-ai-innovation-team/sdg_hub/tree/main/src/sdg_hub/configs/"
    click Prompts "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/src/sdg_hub/prompts.py"
    click Checkpointer "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/src/sdg_hub/checkpointer.py"
    click Dataset "https://github.com/red-hat-ai-innovation-team/sdg_hub/blob/main/src/sdg_hub/utils/datautils.py"

    %% Styles
    classDef core fill:#cfe2f3,stroke:#084298,color:#000;
    classDef config fill:#d1e7dd,stroke:#0f5132,color:#000;
    classDef data fill:#fff3cd,stroke:#664d03,color:#000;
    classDef external fill:#e2e3e5,stroke:#6c757d,color:#000;
```
