# swagath18/synthetic-instruction-tuning-dataset architecture

> Architecture diagram of the GitHub repository swagath18/synthetic-instruction-tuning-dataset, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/swagath18/synthetic-instruction-tuning-dataset
- Repository: https://github.com/swagath18/synthetic-instruction-tuning-dataset

## Overview

##### Project type and purpose
- This is an open-source Python tool for generating synthetic instruction–answer datasets to fine-tune LLMs.
- It focuses on multi-topic dataset creation, rule-based quality checking, and logging failures.

##### File structure
- Top-level scripts:
- generate_data.py – orchestrates data generation.
- quality_checker.py – applies rule-based filters.
- utils.py – common parsing, saving, formatting functions.
- Directories:
- data/ – stores generated JSON instruction files.
- reports/ – logs quality-failure entries (e.g., reports/quality_failures.txt).
- Config & dependencies:
- requirements.txt – Python packages.
- .env – holds OPENAI_API_KEY.

##### README insights
- Describes end-to-end flow: read topics, generate instructions via OpenAI API, run quality checks, persist outputs and failures.
- Mentions future scalability (async/multi-threaded, adversarial filtering).

##### Main components & interactions
a. Components:
- CLI Launcher (User invokes `python generate_data.py`)
- Generator Module (generate_data.py)
- Utilities Module (utils.py)
- Quality Checker (quality_checker.py)
- OpenAI API (external service)
- Data Store (data/ directory)
- Reports Store (reports/ directory)
- Environment Config (.env loader)
b. Relationships & data flow:
1. User → CLI Launcher
2. CLI Launcher → Generator Module
3. Generator Module → Env Loader → OpenAI API
4. OpenAI API → Generator Module (raw responses)
5. Generator Module → Quality Checker → pass/fail decision
6. If pass → Utils → write to Data Store
7. If fail → Utils → write to Reports Store
c. Architectural patterns & principles:
- Modular separation of concerns: generation vs. quality vs. I/O.
- Pluggable topics: instructions_*.json files drive topic loops.
- Config-driven external integration (OpenAI via .env).
d. Key technologies: Python 3, OpenAI API, JSON file I/O.

##### Clean flow for README image
- Title: “Dataset Generation Pipeline”
- Top row:
[User CLI] → [generate_data.py]
- Second row: beneath generate_data.py:
[Env Loader (.env)] → [OpenAI API] → (response)
- Third row:
[quality_checker.py]
- Bottom row:
┌────────────┐   ┌─────────────┐
│ data/ JSON │   │ reports/ txt│
└────────────┘   └─────────────┘

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Nodes
    User["User CLI"] 
    Gen["Generator Module<br/>(generate_data.py)"]:::code_module
    Env["Env Loader<br/>(.env)"]:::config
    API["OpenAI API"]:::external_service
    QC["Quality Checker<br/>(quality_checker.py)"]:::code_module
    Utils["Utilities Module<br/>(utils.py)"]:::code_module
    Data["Data Store<br/>(data/)"]:::storage
    Reports["Reports Store<br/>(reports/)"]:::storage
    Req["requirements.txt"]:::config
    README["README.md"]:::config

    %% Flow
    User -->|invoke| Gen
    Gen -->|load config| Env
    Env -->|provides API key| API
    API -->|returns responses| Gen
    Gen -->|send for validation| QC
    QC -->|pass| Utils
    QC -->|fail| Utils
    Utils -->|write JSON| Data
    Utils -->|log failures| Reports

    %% Supporting files
    Req --> README

    %% Click Events
    click Gen "https://github.com/swagath18/synthetic-instruction-tuning-dataset/blob/main/generate_data.py"
    click Utils "https://github.com/swagath18/synthetic-instruction-tuning-dataset/blob/main/utils.py"
    click QC "https://github.com/swagath18/synthetic-instruction-tuning-dataset/blob/main/quality_checker.py"
    click Data "https://github.com/swagath18/synthetic-instruction-tuning-dataset/tree/main/data/"
    click Req "https://github.com/swagath18/synthetic-instruction-tuning-dataset/blob/main/requirements.txt"
    click README "https://github.com/swagath18/synthetic-instruction-tuning-dataset/blob/main/README.md"

    %% Styles
    classDef code_module fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#333,stroke-width:1px
    classDef external_service fill:#FFA500,stroke:#333,stroke-width:1px
    classDef config fill:#D3D3D3,stroke:#333,stroke-width:1px
```
