# pytorch/ao architecture

> Architecture diagram of the GitHub repository pytorch/ao, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pytorch/ao
- Repository: https://github.com/pytorch/ao

## Overview

##### Project Type and Purpose
- Project Type: Open-source PyTorch extension library.
- Purpose: Provide custom data types (quantization, sparsity, float8, low-bit optimizers) and high-performance kernels for inference and training in PyTorch.

##### File Structure
- Top-Level Modules (all under the torchao/ Python package):
– core/config and utils
– dtypes (float8, uintx, nf4, etc.)
– quantization (pt2e, QAT, GPTQ, HQQ prototypes)
– sparsity (semi-structured 2:4, Marlin, superblock)
– float8 (training and inference recipes)
– optim (8-bit/4-bit/Fp8 optimizers, CPU offload)
– kernel (autotuner, Triton ops)
– csrc (C++/CUDA custom ops)
– experimental (prototype kernels for CPU/AArch64, MPS, ops)
– prototype (research-grade algorithms)
- Utilities and tooling:
– scripts/ (download, convert, packaging helpers)
– examples/ and tutorials/ (end-user code samples)
– docs/ (Sphinx docs, API references)
– packaging/ (build scripts, smoke tests)
– .github/workflows/ (CI for wheels, tests, docs)

##### README for Insights
- Emphasis on composability with torch.compile and FSDP2.
- Two execution branches:
– Pure Python graph transforms (quantize_, swap_linear, convert_to_float8_training)
– Native custom kernels (C++/CUDA, Triton) that torchao loads via PyTorch’s dispatcher.
- Integration points: Hugging Face Transformers, EleutherAI LM-Eval, diffusers, TorchTune.

##### Main Components and Their Interactions
a. User Layer
– Model code or CLI scripts (examples/, tutorials/)
– Calls torchao APIs: quantize_(), swap_linear_with_semi_sparse_linear(), convert_to_float8_training(), optimizer constructors.
b. Python API Layer
– torchao/__init__.py exporting high-level functions.
– Dispatch into submodules: torchao.quantization, torchao.float8, torchao.sparsity, torchao.optim.
c. Core Modules Layer
– Pure-Python implementations: dtype definitions, graph-rewrite passes, fake-quant modules, optimizer wrappers.
– Prototype algorithms under torchao/prototype and torchao/experimental for research.
d. Native Extensions Layer
– C++/CUDA kernels in torchao/csrc (cutlass extensions, fp6_llm, Marlin QQQ, sparse GEMMs).
– Triton ops under torchao/kernel and torchao/prototype/common/triton.
– Built via setup.py and loaded as PyTorch custom ops.
e. External Frameworks & Dependencies
– PyTorch core (torch.compile, FSDP2, dispatch).
– CUDA, cuBLAS, CUTLASS, MPS (macOS).
– Triton compiler.
– Hugging Face Transformers / Diffusers integrations.
f. Hardware Layer
– GPU (CUDA devices), CPU, AArch64 CPU, Apple MPS GPU.
g. Packaging & CI/CD
– setup.py drives build of native extensions.
– GitHub Actions workflows build wheels for multiple CUDA versions, run tests, generate docs.
– Packaging scripts produce PyPI and torch-whl artifacts.

##### Architectural Patterns and Design Principles
- Layered architecture: API → core Python → native extensions → hardware.
- Plugin/dispatcher pattern: register custom ops with torch’s dispatcher to avoid graph breaks in torch.compile.
- Graph-rewrite pattern: pure-Python passes that transform nn.Modules.
- Research vs stable code: clearly separated prototype/experimental namespaces.
- Monorepo with CI gating documentation, benchmarks, multiple hardware targets.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    User["User Code & CLI\n(examples/, tutorials/)"]:::external
    User -->|"calls API"| API

    %% Python API Layer
    subgraph "Python API Layer"
        API["torchao/__init__.py"]:::api
    end

    API -->|"quantize_(), swap_linear(), convert_to_float8_training(), optim constructors"| Core

    %% Core Modules Layer
    subgraph "Core Modules Layer"
        CoreConfig["core/config.py"]:::core
        CoreUtils["utils.py"]:::core
        Dtypes["dtypes/"]:::core
        Quant["quantization/"]:::core
        Spars["sparsity/"]:::core
        F8["float8/"]:::core
        Optims["optim/"]:::core
    end

    CoreConfig --> Core
    CoreUtils --> Core
    Dtypes --> Core
    Quant --> Core
    Spars --> Core
    F8 --> Core
    Optims --> Core

    Core -->|“graph transforms & optimizer wrappers”| PurePython
    Core -->|“register custom ops & call native kernels”| NativeExt

    %% Execution Path Split
    subgraph "Execution Paths"
        subgraph "Pure-Python Execution" 
            PurePython["Fake-quant modules\nGraph rewrite passes\nOptimizer wrappers"]:::core
        end
        subgraph "Native Extensions"
            KernelAuto["autotuner.py"]:::native
            BsrOps["bsr_triton_ops.py"]:::native
            Csrc["csrc/"]:::native
            Prototype["prototype/"]:::native
            Experimental["experimental/"]:::native
        end
    end

    PurePython -->|"runs on"| Hardware
    NativeExt -->|"loads via torch dispatcher"| Hardware

    %% Hardware Layer
    subgraph "Hardware Layer"
        CPU["CPU"]:::hardware
        GPU["GPU (CUDA)"]:::hardware
        MPS["Apple MPS"]:::hardware
        AArch["AArch64 CPU"]:::hardware
    end

    Hardware -.->|“target devices”| GPU
    Hardware -.->|“target devices”| CPU
    Hardware -.->|“target devices”| MPS
    Hardware -.->|“target devices”| AArch

    %% External Integrations
    subgraph "External Frameworks"
        TF["Hugging Face\nTransformers"]:::external
        DF["Diffusers"]:::external
        LME["EleutherAI\nLM-Eval"]:::external
    end

    TF -->|uses torchao APIs| API
    DF -->|uses torchao APIs| API
    LME -->|uses torchao APIs| API

    %% CI/CD & Packaging
    subgraph "CI/CD & Packaging"
        CI["GitHub Actions"]:::cicd
        Setup["setup.py"]:::cicd
        Pack["packaging/"]:::cicd
        Docs["docs/source/"]:::cicd
        PyPI["PyPI Wheels"]:::cicd
    end

    CI -.->|build/test/docs| Setup
    Setup -->|compiles native extensions| NativeExt
    Setup -->|packages wheel| PyPI
    Pack -->|helper scripts| Setup
    CI -.-> Docs
    Docs -->|generate API refs| CI

    %% Click Events
    click API "https://github.com/pytorch/ao/blob/main/torchao/__init__.py"
    click CoreConfig "https://github.com/pytorch/ao/blob/main/torchao/core/config.py"
    click CoreUtils "https://github.com/pytorch/ao/blob/main/torchao/utils.py"
    click Dtypes "https://github.com/pytorch/ao/tree/main/torchao/dtypes/"
    click Quant "https://github.com/pytorch/ao/tree/main/torchao/quantization/"
    click Spars "https://github.com/pytorch/ao/tree/main/torchao/sparsity/"
    click F8 "https://github.com/pytorch/ao/tree/main/torchao/float8/"
    click Optims "https://github.com/pytorch/ao/tree/main/torchao/optim/"
    click KernelAuto "https://github.com/pytorch/ao/blob/main/torchao/kernel/autotuner.py"
    click BsrOps "https://github.com/pytorch/ao/blob/main/torchao/kernel/bsr_triton_ops.py"
    click Csrc "https://github.com/pytorch/ao/tree/main/torchao/csrc/"
    click Prototype "https://github.com/pytorch/ao/tree/main/torchao/prototype/"
    click Experimental "https://github.com/pytorch/ao/tree/main/torchao/experimental/"
    click User "https://github.com/pytorch/ao/tree/main/examples/"
    click User "https://github.com/pytorch/ao/tree/main/tutorials/"
    click Docs "https://github.com/pytorch/ao/tree/main/docs/source/"
    click Setup "https://github.com/pytorch/ao/blob/main/setup.py"
    click Pack "https://github.com/pytorch/ao/tree/main/packaging/"
    click CI "https://github.com/pytorch/ao/tree/main/.github/workflows/"

    %% Styles
    classDef api fill:#cce5ff,stroke:#004085;
    classDef core fill:#cce5ff,stroke:#004085;
    classDef native fill:#ffe5b4,stroke:#ff7300;
    classDef external fill:#d4edda,stroke:#155724;
    classDef hardware fill:#f8f9fa,stroke:#6c757d;
    classDef cicd fill:#e2e0f0,stroke:#4b0082;
```
