# pytorch/torchtitan architecture

> Architecture diagram of the GitHub repository pytorch/torchtitan, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pytorch/torchtitan
- Repository: https://github.com/pytorch/torchtitan

## Overview

##### Project Overview & Purpose
- torchtitan is a library focused on production-level large-scale training of language models (LLMs) using PyTorch.
- Its goal is to showcase native PyTorch distributed training features (e.g., multi-dimensional parallelisms including FSDP2, Tensor Parallel, Pipeline Parallel, and Context Parallel) with a clean, modular, and extendable design.
- The README and file structure reveal that the project is both a research/proof-of-concept codebase (especially for Llama 3.1 models) and a utility library with supporting scripts, experiments, and configuration files for training.

##### Analyzing the File Structure
- The repository is organized into several top-level directories:
- .ci & .github – for CI/CD pipelines, Docker scripts, and automated testing.
- docs – detailed documentation covering performance, debugging, checkpoints, datasets, various parallelisms, and more.
- scripts – helper scripts such as tokenizer downloads, model conversion, memory estimation, and generating model outputs.
- tests – unit and integration tests to ensure reliability.
- torchtitan – core library code split into various submodules:
▪ components – modular components for checkpointing, FLOAT8 support, loss functions, schedulers, etc.
▪ datasets & distributed – support for data handling and distributed training operations.
▪ experiments – folders that tie together experimental setups, including several examples (e.g., “deepseek_v3”, “simple_fsdp”, multimodal experiments).
▪ models – Llama and multimodal models with specific configuration files and helpers for model parallelism.
▪ protocols – definitions that handle model conversion and train specifications.
▪ tools – logging, profiling, and utility functions.
- There’s a clear distinction between core training logic (e.g., torchtitan/train.py) and supporting utilities/test/infrastructure code.

##### Identifying the Main Components
a. Core Training & Model Components:
- Training Engine: The main training loop (see torchtitan/train.py), which integrates model initialization, data loading, and distributed training configurations.
- Model Definitions: Especially for Llama (and its multimodal extension), found under torchtitan/models (with further segmentation for parallelism support components like parallelize_llama.py and pipeline_llama.py).
- Parallelism & Distributed Modules: Distributed training utilities (FSDP2, Tensor Parallel, Pipeline Parallel, Context Parallel) residing in torchtitan/distributed and within various experiment folders.

b. Supporting Components:
- Checkpointing: Components for saving and loading distributed checkpoints (components/checkpoint.py) and distributed data-loading.
- Float8 & Mixed Precision: Provided in torchtitan/components/float8.py.
- Configuration Management: Handling of training configurations via toml files in models/llama/train_configs and configuration logic in torchtitan/config_manager.py.
- Utility Tools: Logging, profiling, and miscellaneous utilities in torchtitan/tools.

c. Auxiliary Modules:
- Experiment modules, which serve as examples and testing grounds for various training schemes.
- Scripts and tests for model conversion, tokenizer downloading (e.g., scripts/download_tokenizer.py), memory estimation, and generation.
- CI/CD & Deployment: Docker configuration in the .ci folder and GitHub workflows that automate testing, linting, and building.

##### Relationships and Interactions
- The training loop (run via run_train.sh) orchestrates the overall process:
- It configures the model based on specified .toml configuration files.
- It sets up the distributed training environment via modules in torchtitan/distributed.
- It loads the model architecture defined in torchtitan/models/llama (or its multimodal counterparts) and applies parallelism – either directly or via helper functions (e.g., parallelize_llama.py and pipeline_llama.py).
- It integrates checkpointing routines (from torchtitan/components/checkpoint.py) and potentially wraps these with Float8 or other mixed-precision configurations.
- External scripts and tools (in “scripts/”) are used to support additional operations such as tokenizer management, model conversion (for DCP format), and performance/memory estimation.
- The CI/CD infrastructure ensures that changes are tested across these components, making the design modular and robust.

##### Architectural Patterns & Design Principles
- Modularization: Separation into distinct directories based on functionality (training, models, components, experiments, distributed logic).
- Configurability & Flexibility: Use of external configuration files (.toml) and modular functions to allow minimal changes to the model code when switching parallelisms or training strategies.
- Composability: Emphasizes composing different parallelism paradigms (FSDP, tensor, pipeline, context) with selective activation checkpointing without hard-coding logic into models.
- Reuse & Extensibility: The design enables integration of new training routines, experiments, or optimizations without affecting the core training loop.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry & Control
    subgraph "Entry & Control"
        TE["Training Engine"]:::core
    end

    %% Model Layer
    subgraph "Model Layer"
        Llama["Llama Model Definitions"]:::core
        Llama_Parallel["Parallelize (Llama)"]:::core
        Llama_Pipeline["Pipeline (Llama)"]:::core
        Llama_Configs["Train Configs (Llama)"]:::config
        Llama_Multimodal["Multimodal Model Definitions"]:::core
        Llama --> Llama_Parallel
        Llama --> Llama_Pipeline
        Llama --> Llama_Configs
    end

    %% Distributed Training Modules
    subgraph "Distributed Training Modules"
        DT["Distributed Training Modules"]:::distributed
        DT_ParallelDims["Parallel Dims"]:::distributed
        DT_Pipeline["Pipeline Module"]:::distributed
        DT_Utils["Distributed Utils"]:::distributed
        DT --> DT_ParallelDims
        DT --> DT_Pipeline
        DT --> DT_Utils
    end

    %% Checkpointing & Mixed Precision
    subgraph "Checkpointing & Mixed Precision"
        CP["Checkpointing"]:::storage
        FP["Float8 & Mixed Precision"]:::storage
    end

    %% Configuration Management
    subgraph "Configuration Management"
        CM["Configuration Management"]:::config
    end

    %% Utility Tools & Profiling
    subgraph "Utility Tools & Profiling"
        UT["Utility Tools"]:::utility
    end

    %% Experiments & Helper Scripts
    subgraph "Experiments & Helper Scripts"
        EX["Experiment Modules"]:::experiments
        HS["Helper Scripts"]:::utility
    end

    %% CI/CD & Deployment
    subgraph "CI/CD & Deployment"
        CI["CI/CD (.ci)"]:::external
        GH["GitHub Workflows (.github)"]:::external
    end

    %% Connections
    TE -->|"initialize_model"| Llama
    TE -->|"initialize_model"| Llama_Multimodal
    TE -->|"set_distributed"| DT
    TE -->|"load_config"| CM
    TE -->|"checkpoint_ops"| CP
    TE -->|"apply_float8"| FP
    TE -->|"log_and_profile"| UT

    Llama -->|"parallelism_helpers"| Llama_Parallel
    Llama -->|"parallelism_helpers"| Llama_Pipeline
    Llama -->|"use_configs"| Llama_Configs

    DT -->|"assist_training"| TE

    CP -->|"save/load_state"| TE
    FP -->|"precision_ops"| TE

    CM -->|"feeds_parameters"| TE

    UT -->|"monitor_training"| TE

    EX -->|"experiment_runs"| TE
    HS -->|"support_ops"| TE

    CI -->|"triggers_tests"| TE
    GH -->|"monitors_ci"| TE

    %% Click Events
    click TE "https://github.com/pytorch/torchtitan/blob/main/torchtitan/train.py"
    click TE "https://github.com/pytorch/torchtitan/blob/main/run_train.sh"

    click Llama "https://github.com/pytorch/torchtitan/tree/main/torchtitan/models/llama"
    click Llama_Parallel "https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/llama/parallelize_llama.py"
    click Llama_Pipeline "https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/llama/pipeline_llama.py"
    click Llama_Configs "https://github.com/pytorch/torchtitan/tree/main/torchtitan/models/llama/train_configs"

    click Llama_Multimodal "https://github.com/pytorch/torchtitan/tree/main/torchtitan/models/llama_multimodal"

    click DT "https://github.com/pytorch/torchtitan/tree/main/torchtitan/distributed"
    click DT_ParallelDims "https://github.com/pytorch/torchtitan/blob/main/torchtitan/distributed/parallel_dims.py"
    click DT_Pipeline "https://github.com/pytorch/torchtitan/blob/main/torchtitan/distributed/pipeline.py"
    click DT_Utils "https://github.com/pytorch/torchtitan/blob/main/torchtitan/distributed/utils.py"

    click CP "https://github.com/pytorch/torchtitan/blob/main/torchtitan/components/checkpoint.py"
    click FP "https://github.com/pytorch/torchtitan/blob/main/torchtitan/components/float8.py"

    click CM "https://github.com/pytorch/torchtitan/blob/main/torchtitan/config_manager.py"
    click CM "https://github.com/pytorch/torchtitan/tree/main/torchtitan/models/llama/train_configs"

    click UT "https://github.com/pytorch/torchtitan/tree/main/torchtitan/tools"
    click UT "https://github.com/pytorch/torchtitan/blob/main/torchtitan/tools/logging.py"
    click UT "https://github.com/pytorch/torchtitan/blob/main/torchtitan/tools/profiling.py"
    click UT "https://github.com/pytorch/torchtitan/blob/main/torchtitan/tools/utils.py"

    click EX "https://github.com/pytorch/torchtitan/tree/main/torchtitan/experiments"
    click HS "https://github.com/pytorch/torchtitan/blob/main/scripts/download_tokenizer.py"
    click HS "https://github.com/pytorch/torchtitan/blob/main/scripts/convert_llama_to_dcp.py"
    click HS "https://github.com/pytorch/torchtitan/tree/main/scripts/estimate"

    click CI "https://github.com/pytorch/torchtitan/blob/main/.ci"
    click GH "https://github.com/pytorch/torchtitan/blob/main/.github"

    %% Styles
    classDef core fill:#f9e79f,stroke:#333,stroke-width:2px;
    classDef distributed fill:#aed6f1,stroke:#333,stroke-width:2px;
    classDef storage fill:#abebc6,stroke:#333,stroke-width:2px;
    classDef config fill:#f5b7b1,stroke:#333,stroke-width:2px;
    classDef utility fill:#d6eaf8,stroke:#333,stroke-width:2px;
    classDef experiments fill:#fcf3cf,stroke:#333,stroke-width:2px;
    classDef external fill:#d5dbdb,stroke:#333,stroke-width:2px;
```
