# lightning-ai/lightning-thunder architecture

> Architecture diagram of the GitHub repository lightning-ai/lightning-thunder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lightning-ai/lightning-thunder
- Repository: https://github.com/lightning-ai/lightning-thunder

## Overview

##### Project Type and Purpose
- Thunder is a compiler that takes PyTorch models and “compiles” them by capturing their computation graphs (traces) and then transforming these traces for optimization (e.g., kernel fusion, quantization, distributed strategies, custom kernels).
- Its purpose is to give PyTorch models “superpowers” by optimizing execution across multiple acceleration strategies (like NVFuser, Triton, and CUDA Graphs) and support distributed training/inference.

##### File Structure for Architectural Clues
- The top-level directories besides configuration files (like .azure, .github, docs, examples, notebooks, and tests) are organized around key modules. The primary source is in the “thunder” directory.
- core: Handles the main optimization routines (like generating, analyzing, and transforming computation traces, managing operations like linear, relu, etc.)
- clang: Likely used for processing or interfacing with C/C++ (potentially for kernel generation or low-level optimizations)
- dynamo: Implements the acquisition of the PyTorch model’s computation graph using dynamic techniques (possibly interacting with torch.compile/dynamo APIs)
- distributed: Contains strategies for handling multi-device/distributed training, tensor parallelism, and includes modules for checkpointing and synchronizing.
- executors: Contains multiple executor implementations for different backends (e.g., NVFuser, apex, torch_autograd) and transformation passes.
- plugins: Houses various extensions that enable optimizations (like quantization, FP8 support, and distributed paradigms)
- transforms: Implements the trace transformations such as fusion, quantization, constant folding, etc.
- There are also directories for “recipes” (higher-level convenience paths for optimizations), “dev_utils” (debugging and profiling utilities), and “tests” (unit/integration tests).

##### Key Components
a. Core Components:
- Model Acquisition: Represented in the “thunder/dynamo” module. This component is responsible for capturing the model’s computation trace.
- Trace Representation and Interpreter: Represented by “thunder/core” and “thunder/trace_interpreter”. This module turns the model into a straight-line Python program (trace) that can be transformed.
- Transformation Pipeline: Under “thunder/transforms” and parts of “thunder/core”. This component performs optimization passes (fusion, quantization, etc.) as indicated in the README.
b. Executor Modules:
- Executors (thunder/executors): Different modules here handle the routing of execution to various backends such as NVFuser, Triton kernels, and even traditional PyTorch eager operations.
c. Distributed and Parallel Strategies:
- Distributed Subsystem: Modules under “thunder/distributed” (including “tensor_parallel” and “transforms/fsdp”, “transforms/ddp”) cover parallelism, synchronization, and checkpointing.
d. Plug-in Architecture:
- Plugins (thunder/plugins): Enables swapping in optimizations like “reduce-overhead” and FP8 support.
e. Integration & Utility:
- Recipes and Dev-Utils: Help users and developers integrate various combinations of optimizations and testing (e.g., benchmark scripts in thunder/benchmarks and analysis tools in thunder/examine).
f. Infrastructure & Supporting Components:
- Docker configs, CI workflows, and documentation (in dockers, .github/workflows, docs) that support development and deployment environments.

##### Relationships and Data Flow
- The acquired trace is processed and transformed through the Chain of Transformation Modules (core and transforms).
- From the transformed trace, the Execution Engine (executors) selects the appropriate backend (NVFuser, Triton, etc.) based on user requirements or plugins passed in during thunder.compile.
- Modules within distributed and plugin directories modify or extend behavior by applying distributed strategies or optimizations.
- Additional utilities (from recipes, benchmarks, and testing components) interact with these core modules through well-defined interfaces.
- Data flow from “User Model” → “Model Acquisition” (capturing trace)
- Transformation pipeline sequences: Trace Representation → Transformation Passes → Prepared Trace
- Execution routing from Prepared Trace to various Executors and back to application-level output.
- Optional plugin paths that modify the transformation or execution behavior.

##### Architectural Patterns and Technologies
- Modular / Layered Architecture: The project cleanly separates concerns (acquisition, transformation, execution, distributed strategies).
- Plugin Architecture: Easily extensible by swapping in and out optimization strategies.
- Source-to-Source Compilation: The transformation of Python code into an optimized trace that is still in Python.
- Distributed computing patterns are present supporting multi-device and parallel optimization strategies.

##### Elements
- Clear labels for each component as described above (i.e., Model Acquisition, Trace Transformation, Executor/Backend, Distributed & Plugin modules).
- Source “User’s PyTorch Model” → “Compiler API (thunder.compile)” → “Dynamo / Trace Capture” → “Transformation Passes” → “Execution Routing (Executors)” → “Optimized Output”
- Annotation of technologies where applicable (e.g., integration with NVFuser, Triton, PyTorch dynamic compilation).

1. Frontend Interface (API entry point via thunder.compile)
2. Model Acquisition & Trace Capture (dynamo modules)
3. Trace Transformation Pipeline (core and transforms components)
4. Executor and Routing Subsystem (executors, distributed modules)
5. Plugin & Extension Framework (plugins, recipe modules)
6. Supporting Infrastructure (CI/CD, Docker, docs)

## Diagram (Mermaid)

```mermaid
graph TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        FE_USER["User's PyTorch Model"]:::frontend
        FE_API["Compiler API (thunder.compile)"]:::frontend
    end

    %% Acquisition Layer
    subgraph "Model Acquisition"
        MA["Trace Capture"]:::acquisition
    end

    %% Trace Processing Layer
    subgraph "Trace Processing"
        TR["Trace Representation"]:::trace
        TR_INT["Trace Interpreter"]:::trace
    end

    %% Transformation Pipeline Layer
    subgraph "Transformation Pipeline"
        T1["Transformation Pipeline"]:::transformation
        CT["Core Transforms"]:::transformation
        TT["Transformed Trace"]:::transformation
    end

    %% Extensions Layer
    subgraph "Optimization Extensions"
        DP["Distributed Strategies"]:::extensions
        PL["Plugin Extensions"]:::extensions
    end

    %% Execution Layer
    subgraph "Execution Engine"
        EX["Executor and Routing"]:::execution
    end

    %% Utility Layer
    subgraph "Integration & Utility"
        UT["Integration & Utility"]:::utility
    end

    %% Output
    OO["Optimized Output"]:::output

    %% Connections: Frontend Flow
    FE_USER -->|"SubmitModel"| FE_API
    FE_API -->|"InvokeCapture"| MA

    %% Connections: Trace Processing Flow
    MA -->|"GenerateTrace"| TR
    TR -->|"Interpret"| TR_INT

    %% Connections: Transformation Pipeline Flow
    TR_INT -->|"ApplyTransforms"| T1
    TR_INT -->|"ApplyCoreTransforms"| CT
    T1 -->|"Merge"| TT
    CT -->|"Merge"| TT

    %% Connections: Extensions influence
    TT -->|"EnableDistributed"| DP
    TT -->|"EnablePlugins"| PL

    %% Connections: Routing to Execution
    TT -->|"PrepareExecution"| EX
    DP -->|"IntegrateDistributed"| EX
    PL -->|"IntegratePlugins"| EX

    %% Integration & Utility interactions
    UT -->|"SupportFrontEnd"| FE_API
    UT -->|"SupportExecution"| EX

    %% Final Output
    EX -->|"ProduceOutput"| OO

    %% Click Events
    click FE_API "https://github.com/lightning-ai/lightning-thunder/blob/main/thunder/dynamo/compiler.py"
    click MA "https://github.com/lightning-ai/lightning-thunder/tree/main/thunder/dynamo"
    click TR "https://github.com/lightning-ai/lightning-thunder/blob/main/thunder/core/trace.py"
    click TR_INT "https://github.com/lightning-ai/lightning-thunder/blob/main/thunder/core/trace_interpreter.py"
    click T1 "https://github.com/lightning-ai/lightning-thunder/tree/main/thunder/transforms"
    click CT "https://github.com/lightning-ai/lightning-thunder/blob/main/thunder/core/transforms.py"
    click DP "https://github.com/lightning-ai/lightning-thunder/tree/main/thunder/distributed"
    click PL "https://github.com/lightning-ai/lightning-thunder/tree/main/thunder/plugins"
    click EX "https://github.com/lightning-ai/lightning-thunder/tree/main/thunder/executors"
    click UT "https://github.com/lightning-ai/lightning-thunder/tree/main/thunder/recipes"
    click UT "https://github.com/lightning-ai/lightning-thunder/tree/main/thunder/dev_utils"

    %% Styles
    classDef frontend fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef acquisition fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef trace fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef transformation fill:#FFA07A,stroke:#000,stroke-width:2px;
    classDef extensions fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef execution fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef utility fill:#D3D3D3,stroke:#000,stroke-width:2px;
    classDef output fill:#F08080,stroke:#000,stroke-width:2px;
```
