How swagath18/synthetic-instruction-tuning-dataset works
Project type and purpose
• This is an open-source Python tool for generating synthetic instruction–answer datasets to fine-tune LLMs.
• It focuses on multi-topic dataset creation, rule-based quality checking, and logging failures.
File structure
• Top-level scripts:
• generate_data.py – orchestrates data generation.
• quality_checker.py – applies rule-based filters.
• utils.py – common parsing, saving, formatting functions.
• Directories:
• data/ – stores generated JSON instruction files.
• reports/ – logs quality-failure entries (e.g., reports/quality_failures.txt).
• Config & dependencies:
• requirements.txt – Python packages.
• .env – holds OPENAI_API_KEY.
README insights
• Describes end-to-end flow: read topics, generate instructions via OpenAI API, run quality checks, persist outputs and failures.
• Mentions future scalability (async/multi-threaded, adversarial filtering).
Main components & interactions
a. Components:
• CLI Launcher (User invokes python generate_data.py)
• Generator Module (generate_data.py)
• Utilities Module (utils.py)
• Quality Checker (quality_checker.py)
• OpenAI API (external service)
• Data Store (data/ directory)
• Reports Store (reports/ directory)
• Environment Config (.env loader)
b. Relationships & data flow:
1. User → CLI Launcher
2. CLI Launcher → Generator Module
3. Generator Module → Env Loader → OpenAI API
4. OpenAI API → Generator Module (raw responses)
5. Generator Module → Quality Checker → pass/fail decision
6. If pass → Utils → write to Data Store
7. If fail → Utils → write to Reports Store
c. Architectural patterns & principles:
• Modular separation of concerns: generation vs. quality vs. I/O.
• Pluggable topics: instructions_*.json files drive topic loops.
• Config-driven external integration (OpenAI via .env).
d. Key technologies: Python 3, OpenAI API, JSON file I/O.
Clean flow for README image
• Title: “Dataset Generation Pipeline”
• Top row:
[User CLI] → [generate_data.py]
• Second row: beneath generate_data.py:
[Env Loader (.env)] → [OpenAI API] → (response)
• Third row:
[quality_checker.py]
• Bottom row:
┌────────────┐ ┌─────────────┐
│ data/ JSON │ │ reports/ txt│
└────────────┘ └─────────────┘