# dao-ailab/flash-attention architecture

> Architecture diagram of the GitHub repository dao-ailab/flash-attention, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dao-ailab/flash-attention
- Repository: https://github.com/dao-ailab/flash-attention

## Overview

FlashAttention is a GPU-accelerated exact-attention library, with CUDA implementations, a Hopper-focused FlashAttention-3 beta, and a CuTeDSL FlashAttention-4 implementation; the README also documents ROCm backends. Its primary workflow is a Python attention call dispatched to a GPU implementation, which computes attention and returns output tensors. The repository additionally includes transformer components and a training workflow. Edges reflect README statements and source-index references; unsampled backend dispatch details are left unspecified.

Main entry points: [`flash_attn/models/gpt.py`](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/models/gpt.py) (GPT models), [`flash_attn/flash_attn_interface.py`](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/flash_attn_interface.py) (Attention interface), [`flash_attn/cute/interface.py`](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/cute/interface.py) (CuTeDSL interface), [`training/run.py`](https://github.com/dao-ailab/flash-attention/blob/main/training/run.py) (Training entry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_attention["Attention APIs"]
  node_fa_api["Attention interface"]
  node_cute_api["CuTeDSL interface<br/>[interface.py]"]
  node_fa3_api["FA3 interface"]
end

subgraph group_kernels["GPU Implementations"]
  node_cuda_api["CUDA bindings<br/>[flash_api.cpp]"]
  node_cuda_fwd["CUDA forward kernel<br/>[flash_fwd_kernel.h]"]
  node_cute_fwd["CuTe forward kernels<br/>[flash_fwd_sm100.py]"]
  node_cute_bwd["CuTe backward kernels<br/>[flash_bwd_sm100.py]"]
  node_hopper_api["Hopper native API<br/>[flash_api.cpp]"]
  node_hopper_mainloop["Hopper mainloop"]
  node_rocm_ck["ROCm CK backend"]
  node_triton_attn["Triton attention"]
end

subgraph group_models["Model Components"]
  node_gpt_models["GPT models<br/>[gpt.py]"]
  node_mha["Multi-head attention<br/>[mha.py]"]
  node_embedding["Embeddings<br/>[embedding.py]"]
  node_generation["Text generation<br/>[generation.py]"]
end

subgraph group_ops["Tensor Operations"]
  node_fused_dense["Fused dense layers<br/>[fused_dense.py]"]
  node_activations["Activations<br/>[activations.py]"]
  node_triton_norm["Triton layer norm<br/>[layer_norm.py]"]
end

subgraph group_training["Training Workflow"]
  node_train_entry["Training entry<br/>[run.py]"]
  node_train["Training loop<br/>[train.py]"]
  node_eval["Evaluation<br/>[eval.py]"]
  node_lm_data["Language data"]
end

node_user(("Python caller"))

node_user -->|"calls"| node_fa_api
node_user -->|"calls"| node_cute_api
node_user -->|"calls"| node_fa3_api
node_fa_api -.->|"dispatches"| node_cuda_api
node_cuda_api -.->|"launches"| node_cuda_fwd
node_cute_api -.->|"dispatches"| node_cute_fwd
node_cute_api -.->|"dispatches"| node_cute_bwd
node_fa3_api -.->|"dispatches"| node_hopper_api
node_hopper_mainloop -.->|"supports"| node_hopper_api
node_fa_api -.->|"supports"| node_rocm_ck
node_fa_api -.->|"supports"| node_triton_attn
node_gpt_models -->|"uses"| node_mha
node_gpt_models -->|"uses"| node_embedding
node_gpt_models -->|"uses"| node_fused_dense
node_gpt_models -->|"uses"| node_activations
node_mha -->|"uses"| node_fused_dense
node_generation -.->|"runs model"| node_gpt_models
node_train_entry -->|"starts"| node_train
node_train_entry -->|"starts"| node_eval
node_train -.->|"loads data"| node_lm_data
node_eval -.->|"loads data"| node_lm_data

click node_fa_api "https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/flash_attn_interface.py"
click node_cute_api "https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/cute/interface.py"
click node_fa3_api "https://github.com/dao-ailab/flash-attention/blob/main/hopper/flash_attn_3/flash_attn_interface.py"
click node_cuda_api "https://github.com/dao-ailab/flash-attention/blob/main/csrc/flash_attn/flash_api.cpp"
click node_cuda_fwd "https://github.com/dao-ailab/flash-attention/blob/main/csrc/flash_attn/src/flash_fwd_kernel.h"
click node_cute_fwd "https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/cute/flash_fwd_sm100.py"
click node_cute_bwd "https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/cute/flash_bwd_sm100.py"
click node_hopper_api "https://github.com/dao-ailab/flash-attention/blob/main/hopper/flash_api.cpp"
click node_hopper_mainloop "https://github.com/dao-ailab/flash-attention/blob/main/hopper/mainloop_fwd_sm90_tma_gmma_ws.hpp"
click node_rocm_ck "https://github.com/dao-ailab/flash-attention/tree/main/csrc/flash_attn_ck"
click node_triton_attn "https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/flash_attn_triton.py"
click node_gpt_models "https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/models/gpt.py"
click node_mha "https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/modules/mha.py"
click node_embedding "https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/modules/embedding.py"
click node_generation "https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/utils/generation.py"
click node_fused_dense "https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/ops/fused_dense.py"
click node_activations "https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/ops/activations.py"
click node_triton_norm "https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/ops/triton/layer_norm.py"
click node_train_entry "https://github.com/dao-ailab/flash-attention/blob/main/training/run.py"
click node_train "https://github.com/dao-ailab/flash-attention/blob/main/training/src/train.py"
click node_eval "https://github.com/dao-ailab/flash-attention/blob/main/training/src/eval.py"
click node_lm_data "https://github.com/dao-ailab/flash-attention/blob/main/training/src/datamodules/language_modeling_hf.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_fa_api,node_cute_api,node_fa3_api toneBlue
class node_cuda_api,node_cuda_fwd,node_cute_fwd,node_cute_bwd,node_hopper_api,node_hopper_mainloop,node_rocm_ck,node_triton_attn toneAmber
class node_gpt_models,node_mha,node_embedding,node_generation toneMint
class node_fused_dense,node_activations,node_triton_norm toneRose
class node_train_entry,node_train,node_eval,node_lm_data,node_user toneIndigo
```

## Components

### Attention APIs

- [Attention interface](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/flash_attn_interface.py): `flash_attn/flash_attn_interface.py`
- [CuTeDSL interface](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/cute/interface.py): `flash_attn/cute/interface.py`
- [FA3 interface](https://github.com/dao-ailab/flash-attention/blob/main/hopper/flash_attn_3/flash_attn_interface.py): `hopper/flash_attn_3/flash_attn_interface.py`

### GPU Implementations

- [CUDA bindings](https://github.com/dao-ailab/flash-attention/blob/main/csrc/flash_attn/flash_api.cpp): `csrc/flash_attn/flash_api.cpp`
- [CUDA forward kernel](https://github.com/dao-ailab/flash-attention/blob/main/csrc/flash_attn/src/flash_fwd_kernel.h): `csrc/flash_attn/src/flash_fwd_kernel.h`
- [CuTe forward kernels](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/cute/flash_fwd_sm100.py): `flash_attn/cute/flash_fwd_sm100.py`
- [CuTe backward kernels](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/cute/flash_bwd_sm100.py): `flash_attn/cute/flash_bwd_sm100.py`
- [Hopper native API](https://github.com/dao-ailab/flash-attention/blob/main/hopper/flash_api.cpp): `hopper/flash_api.cpp`
- [Hopper mainloop](https://github.com/dao-ailab/flash-attention/blob/main/hopper/mainloop_fwd_sm90_tma_gmma_ws.hpp): `hopper/mainloop_fwd_sm90_tma_gmma_ws.hpp`
- [ROCm CK backend](https://github.com/dao-ailab/flash-attention/tree/main/csrc/flash_attn_ck): `csrc/flash_attn_ck`
- [Triton attention](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/flash_attn_triton.py): `flash_attn/flash_attn_triton.py`

### Model Components

- [GPT models](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/models/gpt.py): `flash_attn/models/gpt.py`
- [Multi-head attention](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/modules/mha.py): `flash_attn/modules/mha.py`
- [Embeddings](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/modules/embedding.py): `flash_attn/modules/embedding.py`
- [Text generation](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/utils/generation.py): `flash_attn/utils/generation.py`

### Tensor Operations

- [Fused dense layers](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/ops/fused_dense.py): `flash_attn/ops/fused_dense.py`
- [Activations](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/ops/activations.py): `flash_attn/ops/activations.py`
- [Triton layer norm](https://github.com/dao-ailab/flash-attention/blob/main/flash_attn/ops/triton/layer_norm.py): `flash_attn/ops/triton/layer_norm.py`

### Training Workflow

- [Training entry](https://github.com/dao-ailab/flash-attention/blob/main/training/run.py): `training/run.py`
- [Training loop](https://github.com/dao-ailab/flash-attention/blob/main/training/src/train.py): `training/src/train.py`
- [Evaluation](https://github.com/dao-ailab/flash-attention/blob/main/training/src/eval.py): `training/src/eval.py`
- [Language data](https://github.com/dao-ailab/flash-attention/blob/main/training/src/datamodules/language_modeling_hf.py): `training/src/datamodules/language_modeling_hf.py`

### Other components

- **Python caller**

## Connections

- Python caller → Attention interface: calls (evidence: `README.md`)
- Python caller → CuTeDSL interface: calls (evidence: `README.md`)
- Python caller → FA3 interface: calls (evidence: `README.md`)
- Attention interface → CUDA bindings: dispatches
- CUDA bindings → CUDA forward kernel: launches
- CuTeDSL interface → CuTe forward kernels: dispatches
- CuTeDSL interface → CuTe backward kernels: dispatches
- FA3 interface → Hopper native API: dispatches
- Hopper mainloop → Hopper native API: supports
- Attention interface → ROCm CK backend: supports (evidence: `README.md`)
- Attention interface → Triton attention: supports (evidence: `README.md`)
- GPT models → Multi-head attention: uses (evidence: `flash_attn/models/gpt.py`)
- GPT models → Embeddings: uses (evidence: `flash_attn/models/gpt.py`)
- GPT models → Fused dense layers: uses (evidence: `flash_attn/models/gpt.py`)
- GPT models → Activations: uses (evidence: `flash_attn/models/gpt.py`)
- Multi-head attention → Fused dense layers: uses (evidence: `flash_attn/modules/mha.py`)
- Text generation → GPT models: runs model (evidence: `flash_attn/models/gpt.py`)
- Training entry → Training loop: starts (evidence: `training/run.py`)
- Training entry → Evaluation: starts (evidence: `training/run.py`)
- Training loop → Language data: loads data
- Evaluation → Language data: loads data
