How lightning-ai/lightning-thunder works
Project Type and Purpose
• Thunder is a compiler that takes PyTorch models and “compiles” them by capturing their computation graphs (traces) and then transforming these traces for optimization (e.g., kernel fusion, quantization, distributed strategies, custom kernels).
• Its purpose is to give PyTorch models “superpowers” by optimizing execution across multiple acceleration strategies (like NVFuser, Triton, and CUDA Graphs) and support distributed training/inference.
File Structure for Architectural Clues
• The top-level directories besides configuration files (like .azure, .github, docs, examples, notebooks, and tests) are organized around key modules. The primary source is in the “thunder” directory.
• core: Handles the main optimization routines (like generating, analyzing, and transforming computation traces, managing operations like linear, relu, etc.)
• clang: Likely used for processing or interfacing with C/C++ (potentially for kernel generation or low-level optimizations)
• dynamo: Implements the acquisition of the PyTorch model’s computation graph using dynamic techniques (possibly interacting with torch.compile/dynamo APIs)
• distributed: Contains strategies for handling multi-device/distributed training, tensor parallelism, and includes modules for checkpointing and synchronizing.
• executors: Contains multiple executor implementations for different backends (e.g., NVFuser, apex, torch_autograd) and transformation passes.
• plugins: Houses various extensions that enable optimizations (like quantization, FP8 support, and distributed paradigms)
• transforms: Implements the trace transformations such as fusion, quantization, constant folding, etc.
• There are also directories for “recipes” (higher-level convenience paths for optimizations), “dev_utils” (debugging and profiling utilities), and “tests” (unit/integration tests).
Key Components
a. Core Components:
• Model Acquisition: Represented in the “thunder/dynamo” module. This component is responsible for capturing the model’s computation trace.
• Trace Representation and Interpreter: Represented by “thunder/core” and “thunder/trace_interpreter”. This module turns the model into a straight-line Python program (trace) that can be transformed.
• Transformation Pipeline: Under “thunder/transforms” and parts of “thunder/core”. This component performs optimization passes (fusion, quantization, etc.) as indicated in the README.
b. Executor Modules:
• Executors (thunder/executors): Different modules here handle the routing of execution to various backends such as NVFuser, Triton kernels, and even traditional PyTorch eager operations.
c. Distributed and Parallel Strategies:
• Distributed Subsystem: Modules under “thunder/distributed” (including “tensor_parallel” and “transforms/fsdp”, “transforms/ddp”) cover parallelism, synchronization, and checkpointing.
d. Plug-in Architecture:
• Plugins (thunder/plugins): Enables swapping in optimizations like “reduce-overhead” and FP8 support.
e. Integration & Utility:
• Recipes and Dev-Utils: Help users and developers integrate various combinations of optimizations and testing (e.g., benchmark scripts in thunder/benchmarks and analysis tools in thunder/examine).
f. Infrastructure & Supporting Components:
• Docker configs, CI workflows, and documentation (in dockers, .github/workflows, docs) that support development and deployment environments.
Relationships and Data Flow
• The acquired trace is processed and transformed through the Chain of Transformation Modules (core and transforms).
• From the transformed trace, the Execution Engine (executors) selects the appropriate backend (NVFuser, Triton, etc.) based on user requirements or plugins passed in during thunder.compile.
• Modules within distributed and plugin directories modify or extend behavior by applying distributed strategies or optimizations.
• Additional utilities (from recipes, benchmarks, and testing components) interact with these core modules through well-defined interfaces.
• Data flow from “User Model” → “Model Acquisition” (capturing trace)
• Transformation pipeline sequences: Trace Representation → Transformation Passes → Prepared Trace
• Execution routing from Prepared Trace to various Executors and back to application-level output.
• Optional plugin paths that modify the transformation or execution behavior.
Architectural Patterns and Technologies
• Modular / Layered Architecture: The project cleanly separates concerns (acquisition, transformation, execution, distributed strategies).
• Plugin Architecture: Easily extensible by swapping in and out optimization strategies.
• Source-to-Source Compilation: The transformation of Python code into an optimized trace that is still in Python.
• Distributed computing patterns are present supporting multi-device and parallel optimization strategies.
Elements
• Clear labels for each component as described above (i.e., Model Acquisition, Trace Transformation, Executor/Backend, Distributed & Plugin modules).
• Source “User’s PyTorch Model” → “Compiler API (thunder.compile)” → “Dynamo / Trace Capture” → “Transformation Passes” → “Execution Routing (Executors)” → “Optimized Output”
• Annotation of technologies where applicable (e.g., integration with NVFuser, Triton, PyTorch dynamic compilation).
1. Frontend Interface (API entry point via thunder.compile)
2. Model Acquisition & Trace Capture (dynamo modules)
3. Trace Transformation Pipeline (core and transforms components)
4. Executor and Routing Subsystem (executors, distributed modules)
5. Plugin & Extension Framework (plugins, recipe modules)
6. Supporting Infrastructure (CI/CD, Docker, docs)