# meta-llama/synthetic-data-kit architecture

> Architecture diagram of the GitHub repository meta-llama/synthetic-data-kit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/meta-llama/synthetic-data-kit
- Repository: https://github.com/meta-llama/synthetic-data-kit

## Overview

##### Project type and purpose
- This is an open-source CLI toolkit (Python package) designed to generate and curate high-quality synthetic datasets for fine-tuning large language models (LLMs).
- It implements a four-step pipeline: ingest arbitrary documents, create synthetic examples (QA or Chain-of-Thought), curate/filter them, and save in fine-tuning formats.

##### File structure
- Top-level: documentation (README, DOCS.md), configuration (`configs/config.yaml`, `pyproject.toml`), CLI entrypoint (`synthetic_data_kit/cli.py`), package directory (`synthetic_data_kit/`), and use-case examples under `use-cases/`.
- Inside `synthetic_data_kit/`:
- core/: orchestrates the four commands (`context.py`, `ingest.py`, `create.py`, `curate.py`, `save_as.py`)
- parsers/: adapters that read various file formats (PDF, HTML, DOCX, PPT, TXT, YouTube) and produce plain text
- generators/: logic to turn text into QA pairs or CoT examples
- models/: abstraction over LLM client (vLLM server)
- utils/: shared utilities (config loader, text formatting, llm-processing)
- Configuration: defaults in `configs/config.yaml`, override via CLI flags or user-provided YAML.

##### Extract architectural hints from README
- The four CLI commands map to four core components.
- The system relies on an external vLLM server (“api_base”: `http://localhost:8000/v1`).
- Data flows from raw files → cleaned text → generated examples → curated output → saved as HF/OpenAI formats.

##### Main components and interactions
a. Components
1. CLI Frontend (`synthetic-data-kit` entrypoint)
2. Core Orchestrator (modules in core/)
3. Parsers (parsers/*.py)
4. Generator Engines (generators/*.py)
5. LLM Client (models/llm_client.py talking to vLLM server)
6. Curator (core/curate.py using LLM for scoring)
7. Storage Layers:
– Local FS for intermediate artifacts (data/output, data/generated, data/cleaned)
8. Configuration (configs/config.yaml + override)
9. External Service: vLLM server
b. Data flow and control flow
- CLI → Core.Context loads config → invokes one of:
- ingest: Parsers read file/URL → emit `.txt` in data/output
- create: Generator Engines chunk text → call LLM Client → produce `.json` in data/generated
- curate: LLM Client scores JSON → filter → produce `.json` in data/cleaned
- Config influences generation parameters (temperature, chunk_size, prompts) and LLM endpoint.
c. Architectural patterns/design principles
- Modular pipeline: single-responsibility modules for each stage
- Strategy pattern: interchangeable parsers and generators
- Dependency injection via configuration
- Clear separation of concerns: CLI, core logic, adapters, utilities
d. Key technologies
- Python packaging (`pyproject.toml`)
- Click or argparse for CLI (`cli.py`)
- vLLM server as LLM backend
- YAML for configuration

## Diagram (Mermaid)

```mermaid
flowchart TD
    CLI["CLI Frontend"]:::core
    Context["Core: Context Loader"]:::core
    Ingest["Core: Ingest"]:::core
    Create["Core: Create"]:::core
    Curate["Core: Curate"]:::core
    SaveAs["Core: Save-As"]:::core

    subgraph Parsers
        direction TB
        PDF["PDF Parser"]:::adapter
        HTML["HTML Parser"]:::adapter
        DOCX["DOCX Parser"]:::adapter
        PPT["PPT Parser"]:::adapter
        TXT["TXT Parser"]:::adapter
        YT["YouTube Parser"]:::adapter
    end

    subgraph Generators
        direction TB
        QA["QA Generator"]:::adapter
        CoT["CoT Generator"]:::adapter
    end

    LLM["LLM Client (vLLM)"]:::adapter

    subgraph Utils
        direction TB
        Format["Format Converter"]:::adapter
        LLMProc["LLM Helpers"]:::adapter
        TextUtil["Text Utilities"]:::adapter
        ConfigLoader["Config Loader"]:::config
    end

    ConfigFile["Config: configs/config.yaml"]:::config
    OverrideConfig["Override: synthetic_data_kit/config.yaml"]:::config

    Server["vLLM Server"]:::external

    DataOut["Data Output (data/output)"]:::storage
    DataGen["Data Generated (data/generated)"]:::storage
    DataClean["Data Cleaned (data/cleaned)"]:::storage
    FinalDS["Final Dataset (HF Arrow/JSONL)"]:::storage

    CLI --> Context
    Context --> Ingest
    Context --> Create
    Context --> Curate
    Context --> SaveAs

    ConfigFile --> Context
    OverrideConfig --> Context

    Ingest --> PDF
    Ingest --> HTML
    Ingest --> DOCX
    Ingest --> PPT
    Ingest --> TXT
    Ingest --> YT
    PDF --> DataOut
    HTML --> DataOut
    DOCX --> DataOut
    PPT --> DataOut
    TXT --> DataOut
    YT --> DataOut

    Create --> QA
    Create --> CoT
    QA --> LLM
    CoT --> LLM
    LLM --> DataGen

    Curate --> LLM
    LLM --> DataClean

    SaveAs --> Format
    Format --> FinalDS

    LLM --> Server

    Ingest --> TextUtil
    Create --> LLMProc
    Curate --> LLMProc
    SaveAs --> LLMProc

    click CLI "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/cli.py"
    click Context "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/core/context.py"
    click Ingest "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/core/ingest.py"
    click Create "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/core/create.py"
    click Curate "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/core/curate.py"
    click SaveAs "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/core/save_as.py"
    click PDF "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/parsers/pdf_parser.py"
    click HTML "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/parsers/html_parser.py"
    click DOCX "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/parsers/docx_parser.py"
    click PPT "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/parsers/ppt_parser.py"
    click TXT "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/parsers/txt_parser.py"
    click YT "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/parsers/youtube_parser.py"
    click QA "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/generators/qa_generator.py"
    click CoT "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/generators/cot_generator.py"
    click LLM "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/models/llm_client.py"
    click ConfigFile "https://github.com/meta-llama/synthetic-data-kit/blob/main/configs/config.yaml"
    click OverrideConfig "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/config.yaml"
    click ConfigLoader "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/utils/config.py"
    click Format "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/utils/format_converter.py"
    click LLMProc "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/utils/llm_processing.py"
    click TextUtil "https://github.com/meta-llama/synthetic-data-kit/blob/main/synthetic_data_kit/utils/text.py"
    click pyproject "https://github.com/meta-llama/synthetic-data-kit/blob/main/pyproject.toml"

    classDef core fill:#cce5ff,stroke:#333,stroke-width:1px;
    classDef adapter fill:#d4edda,stroke:#333,stroke-width:1px;
    classDef external fill:#ffe5b4,stroke:#333,stroke-width:1px;
    classDef storage fill:#f0f0f0,stroke:#333,stroke-width:1px;
    classDef config fill:#e2e2e2,stroke:#333,stroke-width:1px;
```
