# iseddik/torchtune architecture

> Architecture diagram of the GitHub repository iseddik/torchtune, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/iseddik/torchtune
- Repository: https://github.com/iseddik/torchtune

## Overview

##### Project Type and Purpose
- torchtune is a Python-based CLI library for authoring and running post-training recipes (SFT, KD, RLHF, QAT) on LLMs. It provides:
- A `tune` CLI entrypoint
- A registry of YAML recipes under `recipes/`
- Model-builder modules (`torchtune/models/…`)
- Data & dataset abstractions (`torchtune/data`, `torchtune/datasets`)
- A training orchestration engine (`torchtune/training`, `torchtune/modules`)
- RLHF support (`torchtune/rlhf`)
- Config parsing and instantiation (`torchtune/config`)
- It integrates with external services (Hugging Face Hub, EleutherAI Eval Harness, W&B, Comet), and supports distributed runs via torchrun.

##### High-Level Directory Analysis
- Top-level CLI (`torchtune/_cli`)
- Config engine (`torchtune/config`) + recipe registry (`torchtune/_recipe_registry.py`)
- Core abstractions: `data`, `datasets`, `generation`, `modules` (neural-net layers & losses)
- Model library: `torchtune/models/{llama2,gemma,…}`
- Training engine: `torchtune/training` + checkpointing
- RLHF engine: `torchtune/rlhf` + `torchtune/dev` for experimental algorithms
- Recipe definitions & examples: `recipes/` & `recipes/configs/…`
- Tests and docs for each component

a. Main Components
- CLI Layer (“tune”): parses subcommands (ls, run, download, validate)
- Config Parser & Instantiator: reads YAML, maps to Python objects (recipes, models, data)
- Recipe Runner: orchestrates end-to-end pipeline (data → model → training loop)
- Model Builders: instantiate pre-defined LLM architectures + tokenizers
- Training Engine: applies configuration to set up optimizer, scheduler, checkpointing, FSDP/RLHF loops
- RLHF Engine: reward models, loss functions (DPO, PPO, GRPO)
- Module Library: fused attention, MoE, PEFT, quantization modules
- Export/Inference: optional conversion & export sub-package
- External Services: Hugging Face Hub, W&B/Comet logging, EleutherAI Eval, torchrun for distribution

b. Relationships & Data Flow
1. User invokes CLI →
2. CLI loads Config Parser →
3. Config Parser instantiates Recipe Runner + components →
4. Recipe Runner calls Dataset Loader → data batches →
5. Recipe Runner calls Model Builder → obtains model + tokenizer →
6. Training Engine runs forward/backward loops using Module Library →
7. RLHF Engine may interpose reward computation →
8. Metrics & checkpoints streamed to External Services →
9. (Optional) Export/Inference sub-pipeline for model conversion

c. Architectural Patterns & Principles
- Plugin-style recipe registry: decouples CLI from recipe implementations
- YAML-driven dependency injection: config module builds all components
- Modular model builders: one directory per supported LLM
- Layered separation: CLI → config → orchestration → compute → I/O
- Use of Torch APIs (compile, FSDP, offload) abstracted in training modules

d. Key Technologies & Frameworks
- Python & PyTorch (core)
- TorchVision, torchao (multimodal & quantization)
- Hugging Face Datasets & Hub
- EleutherAI LM Eval Harness
- Weights & Biases / Comet
- bitsandbytes (8-bit optimizers)
- torchrun / PyTorch Elastic

##### Essential Elements
- Clear component labels matching directory names
- CLI & Config (blue)
- Core orchestration (green)
- Compute modules (orange)
- External systems (gray/cloud)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI & Config Layer
    subgraph "CLI & Config Layer"
        CLI["CLI Layer (tune)"]:::cli
        RecipeRegistry["Recipe Registry"]:::config
        subgraph "Config Engine"
            ConfigParse["_parse.py"]:::config
            ConfigInstantiate["_instantiate.py"]:::config
            ConfigValidate["_validate.py"]:::config
            ConfigUtils["_utils.py"]:::config
            ConfigErrors["_errors.py"]:::config
        end
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        Runner["Recipe Runner"]:::orchestration
        DatasetLoader["Dataset Loader"]:::orchestration
        ModelBuilder["Model Builders"]:::orchestration
        TrainingEngine["Training Engine"]:::orchestration
        RLHFEngine["RLHF Engine"]:::orchestration
        Export["Export/Inference"]:::orchestration
    end

    %% Compute Modules
    subgraph "Compute Modules"
        ModulesLib["Module Library"]:::compute
    end

    %% Persistent Store
    CheckpointStore["Checkpoint Store"]:::persistent

    %% External Services
    subgraph "External Services"
        HFHub["Hugging Face Hub"]:::external
        WnB["W&B / Comet"]:::external
        EleutherAI["EleutherAI Eval"]:::external
        Torchrun["torchrun"]:::external
    end

    %% Data Flow
    CLI -->|"load YAML"| RecipeRegistry
    RecipeRegistry -->|"lookup recipe"| ConfigParse
    ConfigParse --> ConfigInstantiate
    ConfigInstantiate --> Runner
    Runner -->|"load & batch"| DatasetLoader
    DatasetLoader -->|"batches"| Runner
    Runner -->|"build & tokenize"| ModelBuilder
    ModelBuilder --> Runner
    Runner -->|"start training"| TrainingEngine
    TrainingEngine -->|"invoke primitives"| ModulesLib
    TrainingEngine -->|"save weights"| CheckpointStore
    TrainingEngine -->|"emit metrics"| WnB
    TrainingEngine -->|"emit metrics"| EleutherAI
    TrainingEngine -->|"distributed"| Torchrun
    TrainingEngine --> RLHFEngine
    RLHFEngine --> TrainingEngine
    Runner -->|"optional export"| Export
    Export --> HFHub

    %% Click Events
    click CLI "https://github.com/iseddik/torchtune/tree/main/torchtune/_cli/"
    click RecipeRegistry "https://github.com/iseddik/torchtune/blob/main/torchtune/_recipe_registry.py"
    click ConfigParse "https://github.com/iseddik/torchtune/blob/main/torchtune/config/_parse.py"
    click ConfigInstantiate "https://github.com/iseddik/torchtune/blob/main/torchtune/config/_instantiate.py"
    click ConfigValidate "https://github.com/iseddik/torchtune/blob/main/torchtune/config/_validate.py"
    click ConfigUtils "https://github.com/iseddik/torchtune/blob/main/torchtune/config/_utils.py"
    click ConfigErrors "https://github.com/iseddik/torchtune/blob/main/torchtune/config/_errors.py"
    click Runner "https://github.com/iseddik/torchtune/blob/main/torchtune/recipe_interfaces.py"
    click Runner "https://github.com/iseddik/torchtune/blob/main/torchtune/_cli/run.py"
    click Runner "https://github.com/iseddik/torchtune/blob/main/torchtune/_cli/subcommand.py"
    click DatasetLoader "https://github.com/iseddik/torchtune/tree/main/torchtune/data/"
    click DatasetLoader "https://github.com/iseddik/torchtune/tree/main/torchtune/datasets/"
    click DatasetLoader "https://github.com/iseddik/torchtune/tree/main/torchtune/datasets/multimodal/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/clip/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/code_llama2/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/flux/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/gemma/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/gemma2/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/llama2/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/llama3/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/llama4/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/mistral/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/phi3/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/phi4/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/qwen2/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/qwen2_5/"
    click ModelBuilder "https://github.com/iseddik/torchtune/tree/main/torchtune/models/t5/"
    click TrainingEngine "https://github.com/iseddik/torchtune/tree/main/torchtune/training/"
    click TrainingEngine "https://github.com/iseddik/torchtune/tree/main/torchtune/training/checkpointing/"
    click ModulesLib "https://github.com/iseddik/torchtune/blob/main/torchtune/modules/attention.py"
    click ModulesLib "https://github.com/iseddik/torchtune/blob/main/torchtune/modules/feed_forward.py"
    click ModulesLib "https://github.com/iseddik/torchtune/blob/main/torchtune/modules/transformer.py"
    click ModulesLib "https://github.com/iseddik/torchtune/tree/main/torchtune/modules/loss/"
    click ModulesLib "https://github.com/iseddik/torchtune/tree/main/torchtune/modules/model_fusion/"
    click ModulesLib "https://github.com/iseddik/torchtune/tree/main/torchtune/modules/moe/"
    click ModulesLib "https://github.com/iseddik/torchtune/tree/main/torchtune/modules/peft/"
    click ModulesLib "https://github.com/iseddik/torchtune/tree/main/torchtune/modules/low_precision/"
    click ModulesLib "https://github.com/iseddik/torchtune/tree/main/torchtune/modules/_export/"
    click ModulesLib "https://github.com/iseddik/torchtune/tree/main/torchtune/modules/transforms/"
    click RLHFEngine "https://github.com/iseddik/torchtune/tree/main/torchtune/rlhf/"
    click RLHFEngine "https://github.com/iseddik/torchtune/tree/main/torchtune/rlhf/loss/"
    click RLHFEngine "https://github.com/iseddik/torchtune/blob/main/torchtune/rlhf/rewards.py"
    click RLHFEngine "https://github.com/iseddik/torchtune/blob/main/torchtune/rlhf/sequence_processing.py"
    click RLHFEngine "https://github.com/iseddik/torchtune/tree/main/torchtune/dev/rl/"
    click RLHFEngine "https://github.com/iseddik/torchtune/tree/main/torchtune/dev/grpo/"
    click Export "https://github.com/iseddik/torchtune/tree/main/torchtune/modules/_export/"

    %% Styles
    classDef cli fill:#ADD8E6,stroke:#000;
    classDef config fill:#ADD8E6,stroke:#000;
    classDef orchestration fill:#90EE90,stroke:#000;
    classDef compute fill:#FFA500,stroke:#000;
    classDef external fill:#D3D3D3,stroke:#000;
    classDef persistent fill:#B0C4DE,stroke:#000;
```
