How red-hat-ai-innovation-team/sdg_hub works
Project type and purpose
• Type: A Python‐based SDK / toolkit for building “no‐code” synthetic data generation pipelines for LLMs, plus a minimal web UI.
File structure
Top‐level directories:
• src/sdg_hub/ – Core library
– blocks/ (Block implementations: LLMBlock, FilterByValueBlock, util blocks, etc.)
– configs/ (YAML prompt templates and block settings)
– flows/ (Sample flow definitions)
– flow.py, flow_runner.py, pipeline.py, registry.py, checkpointer.py
– prompts.py, sdg.py (CLI entrypoint), logger_config.py
• examples/ – Jupyter notebooks and sample flows demonstrating usage
• web_interface/ – A Flask (or lightweight) web app (app.py) serving a JS/HTML front end to run flows interactively
• tests/ – Unit tests for blocks, flows, runner
• scripts/ – Dev helpers (e.g., lint wrappers)
Configuration and CI: pyproject.toml, GitHub Actions workflows, pre-commit, linters.
Insights from the README
• Core concepts: Blocks, Flows, Prompts (YAML), Checkpoints, HuggingFace Datasets.
• Design principles: Modular, no-code, scalable, checkpoint-resumable.
a. Main components
1. User Interface Layer
– CLI (sdg.py)
– Web UI (web_interface/app.py + JS/CSS/HTML)
2. Orchestration Layer
– Flow Loader: Reads YAML flow file
– FlowRunner + Registry: Maps block_type → Block class
– Pipeline: Constructs ordered graph of Blocks
3. Execution Layer
– Blocks: Instances of LLMBlock, FilterByValueBlock, IterBlock, utility blocks
– Config Loader: Loads block/prompt YAML under configs/
– Checkpointer: Persists intermediate HuggingFace datasets to disk
4. Data Layer
5. External Services
– LLM API endpoints (HuggingFace Hub, OpenAI, or other model endpoints)
b. Relationships & Interactions
1. UI triggers the FlowRunner, passing the path to a YAML flow.
2. FlowRunner uses Registry to instantiate Block objects in order.
3. Pipeline executes each Block in sequence or nested iterations:
– Blocks load their prompt/config YAML (via configs/)
– LLMBlock issues HTTP calls to an LLM service and awaits a response
– Other blocks transform/filter the dataset in memory
4. Final dataset is returned to the CLI (printed or written) or sent back to the Web UI for display.
c. Architectural patterns & principles
– Registry/Factory pattern for block instantiation
– Pipeline pattern for chaining computational steps
– Config‐driven design (no code changes needed to build new flows)
– Modularity & reusability: blocks are self-contained, interchangeable units
– Checkpointing for fault tolerance and resumability
d. Key technologies & libraries
– HTTP clients for LLM APIs (requests or huggingface-hub)
– Flask (or lightweight WSGI) for web_interface/app.py
– JavaScript/CSS/HTML for the web front end
Detail
• Individual modules (e.g., flow_runner.py, registry.py, blocks/llmblock.py) only if necessary for clarity.
• Essential layers: Config → Engine → Data → External.
• Do not over-abstract: include the Checkpointer and HuggingFace Datasets as first-class components.