# tridao/flash-attention-wheels architecture

> Architecture diagram of the GitHub repository tridao/flash-attention-wheels, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tridao/flash-attention-wheels
- Repository: https://github.com/tridao/flash-attention-wheels

## Overview

FlashAttention is a PyTorch-facing CUDA library for exact, memory-efficient attention. The principal workflow accepts packed or separate Q/K/V tensors, dispatches GPU forward attention, and uses a custom autograd backward path to produce gradients. Documented capabilities include variable-length and grouped-query attention, causal and local windows, dropout, split-KV execution, and incremental KV-cache updates with rotary embeddings. The graph includes the Python interface, C++ dispatch, and CUDA kernel layers; several links rely on the README or SOURCE INDEX because the excerpts are partial.

Main entry points: [`csrc/flash_attn/flash_api.cpp`](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/flash_api.cpp) (Tensor binding), [`csrc/flash_attn/src/flash_fwd_kernel.h`](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/flash_fwd_kernel.h) (Forward attention), [`csrc/flash_attn/src/flash_bwd_kernel.h`](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/flash_bwd_kernel.h) (Backward gradients), [`csrc/flash_attn/src/flash_fwd_launch_template.h`](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/flash_fwd_launch_template.h) (Forward launch)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Python API"]
  node_python_api["Attention interface"]
  node_package["Package exports<br/>[__init__.py]"]
end

subgraph group_dispatch["CUDA Dispatch"]
  node_binding["Tensor binding<br/>[flash_api.cpp]"]
  node_params["Kernel parameters<br/>[flash.h]"]
  node_fwd_dispatch["Forward dispatch<br/>[flash_api.cpp]"]
  node_bwd_dispatch["Backward dispatch<br/>[flash_api.cpp]"]
end

subgraph group_compute["GPU Compute"]
  node_fwd_launch["Forward launch"]
  node_fwd_kernel["Forward attention<br/>[flash_fwd_kernel.h]"]
  node_bwd_launch["Backward launch"]
  node_bwd_kernel["Backward gradients<br/>[flash_bwd_kernel.h]"]
  node_traits["Kernel traits<br/>[kernel_traits.h]"]
  node_block_info["Sequence metadata<br/>[block_info.h]"]
  node_softmax["Softmax math<br/>[softmax.h]"]
  node_utils["GPU utilities<br/>[utils.h]"]
  node_switches["Specialization switches<br/>[static_switch.h]"]
end

node_caller(("PyTorch caller"))

node_caller -->|"calls attention"| node_python_api
node_package -->|"exports"| node_python_api
node_python_api -.->|"invokes CUDA"| node_binding
node_binding -->|"fills parameters"| node_params
node_binding -.->|"dispatches forward"| node_fwd_dispatch
node_binding -.->|"dispatches backward"| node_bwd_dispatch
node_fwd_dispatch -.->|"launches kernels"| node_fwd_launch
node_bwd_dispatch -.->|"launches kernels"| node_bwd_launch
node_fwd_launch -->|"launches"| node_fwd_kernel
node_bwd_launch -->|"launches"| node_bwd_kernel
node_fwd_kernel -->|"uses layouts"| node_traits
node_bwd_kernel -->|"uses layouts"| node_traits
node_fwd_kernel -->|"uses sequence metadata"| node_block_info
node_bwd_kernel -->|"uses sequence metadata"| node_block_info
node_fwd_kernel -->|"computes softmax"| node_softmax
node_bwd_kernel -->|"uses softmax math"| node_softmax
node_softmax -->|"uses reductions"| node_utils
node_fwd_launch -->|"selects variants"| node_switches
node_bwd_launch -->|"selects variants"| node_switches

click node_python_api "https://github.com/tridao/flash-attention-wheels/blob/main/flash_attn/flash_attn_interface.py"
click node_package "https://github.com/tridao/flash-attention-wheels/blob/main/flash_attn/__init__.py"
click node_binding "https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/flash_api.cpp"
click node_params "https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/flash.h"
click node_fwd_dispatch "https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/flash_api.cpp"
click node_bwd_dispatch "https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/flash_api.cpp"
click node_fwd_launch "https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/flash_fwd_launch_template.h"
click node_fwd_kernel "https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/flash_fwd_kernel.h"
click node_bwd_launch "https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/flash_bwd_launch_template.h"
click node_bwd_kernel "https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/flash_bwd_kernel.h"
click node_traits "https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/kernel_traits.h"
click node_block_info "https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/block_info.h"
click node_softmax "https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/softmax.h"
click node_utils "https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/utils.h"
click node_switches "https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/static_switch.h"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_python_api,node_package toneBlue
class node_binding,node_params,node_fwd_dispatch,node_bwd_dispatch toneAmber
class node_fwd_launch,node_fwd_kernel,node_bwd_launch,node_bwd_kernel,node_traits,node_block_info,node_softmax,node_utils,node_switches toneMint
class node_caller toneIndigo
```

## Components

### Python API

- [Attention interface](https://github.com/tridao/flash-attention-wheels/blob/main/flash_attn/flash_attn_interface.py): `flash_attn/flash_attn_interface.py`
- [Package exports](https://github.com/tridao/flash-attention-wheels/blob/main/flash_attn/__init__.py): `flash_attn/__init__.py`

### CUDA Dispatch

- [Tensor binding](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/flash_api.cpp): `csrc/flash_attn/flash_api.cpp`
- [Kernel parameters](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/flash.h): `csrc/flash_attn/src/flash.h`
- [Forward dispatch](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/flash_api.cpp): `csrc/flash_attn/flash_api.cpp`
- [Backward dispatch](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/flash_api.cpp): `csrc/flash_attn/flash_api.cpp`

### GPU Compute

- [Forward launch](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/flash_fwd_launch_template.h): `csrc/flash_attn/src/flash_fwd_launch_template.h`
- [Forward attention](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/flash_fwd_kernel.h): `csrc/flash_attn/src/flash_fwd_kernel.h`
- [Backward launch](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/flash_bwd_launch_template.h): `csrc/flash_attn/src/flash_bwd_launch_template.h`
- [Backward gradients](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/flash_bwd_kernel.h): `csrc/flash_attn/src/flash_bwd_kernel.h`
- [Kernel traits](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/kernel_traits.h): `csrc/flash_attn/src/kernel_traits.h`
- [Sequence metadata](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/block_info.h): `csrc/flash_attn/src/block_info.h`
- [Softmax math](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/softmax.h): `csrc/flash_attn/src/softmax.h`
- [GPU utilities](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/utils.h): `csrc/flash_attn/src/utils.h`
- [Specialization switches](https://github.com/tridao/flash-attention-wheels/blob/main/csrc/flash_attn/src/static_switch.h): `csrc/flash_attn/src/static_switch.h`

### Other components

- **PyTorch caller**

## Connections

- PyTorch caller → Attention interface: calls attention (evidence: `README.md`)
- Package exports → Attention interface: exports (evidence: `flash_attn/__init__.py`)
- Attention interface → Tensor binding: invokes CUDA
- Tensor binding → Kernel parameters: fills parameters (evidence: `csrc/flash_attn/flash_api.cpp`)
- Tensor binding → Forward dispatch: dispatches forward (evidence: `csrc/flash_attn/flash_api.cpp`)
- Tensor binding → Backward dispatch: dispatches backward (evidence: `csrc/flash_attn/flash_api.cpp`)
- Forward dispatch → Forward launch: launches kernels
- Backward dispatch → Backward launch: launches kernels
- Forward launch → Forward attention: launches (evidence: `csrc/flash_attn/src/flash_fwd_launch_template.h`)
- Backward launch → Backward gradients: launches (evidence: `csrc/flash_attn/src/flash_bwd_launch_template.h`)
- Forward attention → Kernel traits: uses layouts (evidence: `csrc/flash_attn/src/flash_fwd_kernel.h`)
- Backward gradients → Kernel traits: uses layouts (evidence: `csrc/flash_attn/src/flash_bwd_kernel.h`)
- Forward attention → Sequence metadata: uses sequence metadata (evidence: `csrc/flash_attn/src/flash_fwd_kernel.h`)
- Backward gradients → Sequence metadata: uses sequence metadata (evidence: `csrc/flash_attn/src/flash_bwd_kernel.h`)
- Forward attention → Softmax math: computes softmax (evidence: `csrc/flash_attn/src/flash_fwd_kernel.h`)
- Backward gradients → Softmax math: uses softmax math (evidence: `csrc/flash_attn/src/flash_bwd_kernel.h`)
- Softmax math → GPU utilities: uses reductions (evidence: `csrc/flash_attn/src/softmax.h`)
- Forward launch → Specialization switches: selects variants (evidence: `csrc/flash_attn/src/flash_fwd_launch_template.h`)
- Backward launch → Specialization switches: selects variants (evidence: `csrc/flash_attn/src/flash_bwd_launch_template.h`)
