How tridao/flash-attention-wheels works
FlashAttention is a PyTorch-facing CUDA library for exact, memory-efficient attention. The principal workflow accepts packed or separate Q/K/V tensors, dispatches GPU forward attention, and uses a custom autograd backward path to produce gradients. Documented capabilities include variable-length and grouped-query attention, causal and local windows, dropout, split-KV execution, and incremental KV-cache updates with rotary embeddings. The graph includes the Python interface, C++ dispatch, and CUDA kernel layers; several links rely on the README or SOURCE INDEX because the excerpts are partial.
Main entry points: flash_api.cpp Tensor binding · flash_fwd_kernel.h Forward attention · flash_bwd_kernel.h Backward gradients · flash_fwd_launch_template.h Forward launch
Components 16
Python API
- Attention interface
flash_attn/flash_attn_interface.py - Package exports
flash_attn/__init__.py
CUDA Dispatch
- Tensor binding
csrc/flash_attn/flash_api.cpp - Kernel parameters
csrc/flash_attn/src/flash.h - Forward dispatch
csrc/flash_attn/flash_api.cpp - Backward dispatch
csrc/flash_attn/flash_api.cpp
GPU Compute
- Forward launch
csrc/flash_attn/src/flash_fwd_launch_template.h - Forward attention
csrc/flash_attn/src/flash_fwd_kernel.h - Backward launch
csrc/flash_attn/src/flash_bwd_launch_template.h - Backward gradients
csrc/flash_attn/src/flash_bwd_kernel.h - Kernel traits
csrc/flash_attn/src/kernel_traits.h - Sequence metadata
csrc/flash_attn/src/block_info.h - Softmax math
csrc/flash_attn/src/softmax.h - GPU utilities
csrc/flash_attn/src/utils.h - Specialization switches
csrc/flash_attn/src/static_switch.h
Other components
- PyTorch caller
Connections 19
- PyTorch caller → Attention interface: calls attention
README.md - Package exports → Attention interface: exports
flash_attn/__init__.py - Attention interface → Tensor binding: invokes CUDA
- Tensor binding → Kernel parameters: fills parameters
csrc/flash_attn/flash_api.cpp - Tensor binding → Forward dispatch: dispatches forward
csrc/flash_attn/flash_api.cpp - Tensor binding → Backward dispatch: dispatches backward
csrc/flash_attn/flash_api.cpp - Forward dispatch → Forward launch: launches kernels
- Backward dispatch → Backward launch: launches kernels
- Forward launch → Forward attention: launches
csrc/flash_attn/src/flash_fwd_launch_template.h - Backward launch → Backward gradients: launches
csrc/flash_attn/src/flash_bwd_launch_template.h - Forward attention → Kernel traits: uses layouts
csrc/flash_attn/src/flash_fwd_kernel.h - Backward gradients → Kernel traits: uses layouts
csrc/flash_attn/src/flash_bwd_kernel.h - Forward attention → Sequence metadata: uses sequence metadata
csrc/flash_attn/src/flash_fwd_kernel.h - Backward gradients → Sequence metadata: uses sequence metadata
csrc/flash_attn/src/flash_bwd_kernel.h - Forward attention → Softmax math: computes softmax
csrc/flash_attn/src/flash_fwd_kernel.h - Backward gradients → Softmax math: uses softmax math
csrc/flash_attn/src/flash_bwd_kernel.h - Softmax math → GPU utilities: uses reductions
csrc/flash_attn/src/softmax.h - Forward launch → Specialization switches: selects variants
csrc/flash_attn/src/flash_fwd_launch_template.h - Backward launch → Specialization switches: selects variants
csrc/flash_attn/src/flash_bwd_launch_template.h