How dao-ailab/flash-attention works
FlashAttention is a GPU-accelerated exact-attention library, with CUDA implementations, a Hopper-focused FlashAttention-3 beta, and a CuTeDSL FlashAttention-4 implementation; the README also documents ROCm backends. Its primary workflow is a Python attention call dispatched to a GPU implementation, which computes attention and returns output tensors. The repository additionally includes transformer components and a training workflow. Edges reflect README statements and source-index references; unsampled backend dispatch details are left unspecified.
Main entry points: gpt.py GPT models · flash_attn_interface.py Attention interface · interface.py CuTeDSL interface · run.py Training entry
Components 23
Attention APIs
- Attention interface
flash_attn/flash_attn_interface.py - CuTeDSL interface
flash_attn/cute/interface.py - FA3 interface
hopper/flash_attn_3/flash_attn_interface.py
GPU Implementations
- CUDA bindings
csrc/flash_attn/flash_api.cpp - CUDA forward kernel
csrc/flash_attn/src/flash_fwd_kernel.h - CuTe forward kernels
flash_attn/cute/flash_fwd_sm100.py - CuTe backward kernels
flash_attn/cute/flash_bwd_sm100.py - Hopper native API
hopper/flash_api.cpp - Hopper mainloop
hopper/mainloop_fwd_sm90_tma_gmma_ws.hpp - ROCm CK backend
csrc/flash_attn_ck - Triton attention
flash_attn/flash_attn_triton.py
Model Components
- GPT models
flash_attn/models/gpt.py - Multi-head attention
flash_attn/modules/mha.py - Embeddings
flash_attn/modules/embedding.py - Text generation
flash_attn/utils/generation.py
Tensor Operations
- Fused dense layers
flash_attn/ops/fused_dense.py - Activations
flash_attn/ops/activations.py - Triton layer norm
flash_attn/ops/triton/layer_norm.py
Training Workflow
- Training entry
training/run.py - Training loop
training/src/train.py - Evaluation
training/src/eval.py - Language data
training/src/datamodules/language_modeling_hf.py
Other components
- Python caller
Connections 21
- Python caller → Attention interface: calls
README.md - Python caller → CuTeDSL interface: calls
README.md - Python caller → FA3 interface: calls
README.md - Attention interface → CUDA bindings: dispatches
- CUDA bindings → CUDA forward kernel: launches
- CuTeDSL interface → CuTe forward kernels: dispatches
- CuTeDSL interface → CuTe backward kernels: dispatches
- FA3 interface → Hopper native API: dispatches
- Hopper mainloop → Hopper native API: supports
- Attention interface → ROCm CK backend: supports
README.md - Attention interface → Triton attention: supports
README.md - GPT models → Multi-head attention: uses
flash_attn/models/gpt.py - GPT models → Embeddings: uses
flash_attn/models/gpt.py - GPT models → Fused dense layers: uses
flash_attn/models/gpt.py - GPT models → Activations: uses
flash_attn/models/gpt.py - Multi-head attention → Fused dense layers: uses
flash_attn/modules/mha.py - Text generation → GPT models: runs model
flash_attn/models/gpt.py - Training entry → Training loop: starts
training/run.py - Training entry → Evaluation: starts
training/run.py - Training loop → Language data: loads data
- Evaluation → Language data: loads data