How pytorch/ao works
Project Type and Purpose
• Project Type: Open-source PyTorch extension library.
• Purpose: Provide custom data types (quantization, sparsity, float8, low-bit optimizers) and high-performance kernels for inference and training in PyTorch.
File Structure
• Top-Level Modules (all under the torchao/ Python package):
– core/config and utils
– dtypes (float8, uintx, nf4, etc.)
– quantization (pt2e, QAT, GPTQ, HQQ prototypes)
– sparsity (semi-structured 2:4, Marlin, superblock)
– float8 (training and inference recipes)
– optim (8-bit/4-bit/Fp8 optimizers, CPU offload)
– kernel (autotuner, Triton ops)
– csrc (C++/CUDA custom ops)
– experimental (prototype kernels for CPU/AArch64, MPS, ops)
– prototype (research-grade algorithms)
• Utilities and tooling:
– scripts/ (download, convert, packaging helpers)
– examples/ and tutorials/ (end-user code samples)
– docs/ (Sphinx docs, API references)
– packaging/ (build scripts, smoke tests)
– .github/workflows/ (CI for wheels, tests, docs)
README for Insights
• Emphasis on composability with torch.compile and FSDP2.
• Two execution branches:
– Pure Python graph transforms (quantize_, swap_linear, convert_to_float8_training)
– Native custom kernels (C++/CUDA, Triton) that torchao loads via PyTorch’s dispatcher.
• Integration points: Hugging Face Transformers, EleutherAI LM-Eval, diffusers, TorchTune.
Main Components and Their Interactions
a. User Layer
– Model code or CLI scripts (examples/, tutorials/)
– Calls torchao APIs: quantize_(), swap_linear_with_semi_sparse_linear(), convert_to_float8_training(), optimizer constructors.
b. Python API Layer
– torchao/__init__.py exporting high-level functions.
– Dispatch into submodules: torchao.quantization, torchao.float8, torchao.sparsity, torchao.optim.
c. Core Modules Layer
– Pure-Python implementations: dtype definitions, graph-rewrite passes, fake-quant modules, optimizer wrappers.
– Prototype algorithms under torchao/prototype and torchao/experimental for research.
d. Native Extensions Layer
– C++/CUDA kernels in torchao/csrc (cutlass extensions, fp6_llm, Marlin QQQ, sparse GEMMs).
– Triton ops under torchao/kernel and torchao/prototype/common/triton.
– Built via setup.py and loaded as PyTorch custom ops.
e. External Frameworks & Dependencies
– PyTorch core (torch.compile, FSDP2, dispatch).
– CUDA, cuBLAS, CUTLASS, MPS (macOS).
– Triton compiler.
– Hugging Face Transformers / Diffusers integrations.
f. Hardware Layer
– GPU (CUDA devices), CPU, AArch64 CPU, Apple MPS GPU.
g. Packaging & CI/CD
– setup.py drives build of native extensions.
– GitHub Actions workflows build wheels for multiple CUDA versions, run tests, generate docs.
– Packaging scripts produce PyPI and torch-whl artifacts.
Architectural Patterns and Design Principles
• Layered architecture: API → core Python → native extensions → hardware.
• Plugin/dispatcher pattern: register custom ops with torch’s dispatcher to avoid graph breaks in torch.compile.
• Graph-rewrite pattern: pure-Python passes that transform nn.Modules.
• Research vs stable code: clearly separated prototype/experimental namespaces.
• Monorepo with CI gating documentation, benchmarks, multiple hardware targets.