How woct0rdho/sageattention works
SageAttention is a GPU-accelerated attention package with standard int8-based attention paths and a separate Blackwell FP4 path. The main workflow accepts query, key and value tensors through the Python API, applies quantization and attention kernels, and returns the output (optionally with LSE). The README also documents variable-length and causal attention and support across several GPU generations. Source excerpts show key calls, but not every dispatch or extension binding, so some implementation links are omitted.
Main entry points: core.py Attention dispatch · kernel_ws.h Blackwell kernel · api.py Blackwell API · __init__.py Public API
Components 23
Python API
- Public API
sageattention/__init__.py - Attention dispatch
sageattention/core.py
Standard Attention
- Quantization API
sageattention/quant.py - Block quantization
sageattention/triton/quant_per_block.py - Varlen quantization
sageattention/triton/quant_per_block_varlen.py - Per-thread quantization
sageattention/triton/quant_per_thread.py - Block attention
sageattention/triton/attn_qk_int8_per_block.py - Causal attention
sageattention/triton/attn_qk_int8_per_block_causal.py - Varlen attention
sageattention/triton/attn_qk_int8_block_varlen.py - Causal varlen attention
sageattention/triton/attn_qk_int8_per_block_causal_varlen.py
CUDA Kernels
- Fused CUDA ops
csrc/fused/fused.h - CUDA bindings
csrc/fused/pybind.cpp - SM80 attention
csrc/qattn/attn_cuda_sm80.h - SM89 attention
csrc/qattn/attn_cuda_sm89.h - SM90 attention
csrc/qattn/attn_cuda_sm90.h
Blackwell FP4
- Blackwell API
sageattention3_blackwell/sageattn3/api.py - Blackwell kernel
sageattention3_blackwell/csrc/blackwell/kernel_ws.h - TMA mainloop
sageattention3_blackwell/csrc/blackwell/mainloop_tma_ws.h - TMA epilogue
sageattention3_blackwell/csrc/blackwell/epilogue_tma_ws.h - Fused softmax
sageattention3_blackwell/csrc/blackwell/softmax_fused.h - Pipeline barriers
sageattention3_blackwell/csrc/blackwell/named_barrier.h
Other components
- Attention caller
- Attention output
Connections 23
- Attention caller → Public API: calls
README.md - Public API → Attention dispatch: exports
sageattention/__init__.py - Attention dispatch → Quantization API: uses
sageattention/core.py - Attention dispatch → Block quantization: uses
sageattention/core.py - Attention dispatch → Varlen quantization: uses
sageattention/core.py - Attention dispatch → Per-thread quantization: uses
sageattention/core.py - Attention dispatch → Block attention: uses
sageattention/core.py - Attention dispatch → Causal attention: uses
sageattention/core.py - Attention dispatch → Varlen attention: uses
sageattention/core.py - Attention dispatch → Causal varlen attention: uses
sageattention/core.py - Attention dispatch → SM80 attention: dispatches
- Attention dispatch → SM89 attention: dispatches
- Attention dispatch → SM90 attention: dispatches
- Quantization API → Fused CUDA ops: calls
sageattention/quant.py - CUDA bindings → Fused CUDA ops: binds
csrc/fused/pybind.cpp - Attention dispatch → Attention output: returns
sageattention/core.py - Attention caller → Blackwell API: calls
- Blackwell API → Blackwell kernel: invokes
- Blackwell kernel → TMA mainloop: uses
sageattention3_blackwell/csrc/blackwell/kernel_ws.h - Blackwell kernel → TMA epilogue: uses
sageattention3_blackwell/csrc/blackwell/kernel_ws.h - Blackwell kernel → Fused softmax: uses
sageattention3_blackwell/csrc/blackwell/kernel_ws.h - Blackwell kernel → Pipeline barriers: uses
sageattention3_blackwell/csrc/blackwell/kernel_ws.h - Blackwell API → Attention output: returns
sageattention3_blackwell/sageattn3/api.py