How meta-llama/synthetic-data-kit works
Project type and purpose
• This is an open-source CLI toolkit (Python package) designed to generate and curate high-quality synthetic datasets for fine-tuning large language models (LLMs).
• It implements a four-step pipeline: ingest arbitrary documents, create synthetic examples (QA or Chain-of-Thought), curate/filter them, and save in fine-tuning formats.
File structure
• Top-level: documentation (README, DOCS.md), configuration (configs/config.yaml, pyproject.toml), CLI entrypoint (synthetic_data_kit/cli.py), package directory (synthetic_data_kit/), and use-case examples under use-cases/.
• Inside synthetic_data_kit/:
• core/: orchestrates the four commands (context.py, ingest.py, create.py, curate.py, save_as.py)
• parsers/: adapters that read various file formats (PDF, HTML, DOCX, PPT, TXT, YouTube) and produce plain text
• generators/: logic to turn text into QA pairs or CoT examples
• models/: abstraction over LLM client (vLLM server)
• utils/: shared utilities (config loader, text formatting, llm-processing)
• Configuration: defaults in configs/config.yaml, override via CLI flags or user-provided YAML.
Extract architectural hints from README
• The four CLI commands map to four core components.
• The system relies on an external vLLM server (“api_base”: http://localhost:8000/v1).
• Data flows from raw files → cleaned text → generated examples → curated output → saved as HF/OpenAI formats.
Main components and interactions
a. Components
1. CLI Frontend (synthetic-data-kit entrypoint)
2. Core Orchestrator (modules in core/)
3. Parsers (parsers/*.py)
4. Generator Engines (generators/*.py)
5. LLM Client (models/llm_client.py talking to vLLM server)
6. Curator (core/curate.py using LLM for scoring)
7. Storage Layers:
– Local FS for intermediate artifacts (data/output, data/generated, data/cleaned)
8. Configuration (configs/config.yaml + override)
9. External Service: vLLM server
b. Data flow and control flow
• CLI → Core.Context loads config → invokes one of:
• ingest: Parsers read file/URL → emit .txt in data/output
• create: Generator Engines chunk text → call LLM Client → produce .json in data/generated
• curate: LLM Client scores JSON → filter → produce .json in data/cleaned
• Config influences generation parameters (temperature, chunk_size, prompts) and LLM endpoint.
c. Architectural patterns/design principles
• Modular pipeline: single-responsibility modules for each stage
• Strategy pattern: interchangeable parsers and generators
• Dependency injection via configuration
• Clear separation of concerns: CLI, core logic, adapters, utilities
d. Key technologies
• Python packaging (pyproject.toml)
• Click or argparse for CLI (cli.py)
• vLLM server as LLM backend
• YAML for configuration