# woct0rdho/sageattention architecture

> Architecture diagram of the GitHub repository woct0rdho/sageattention, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/woct0rdho/sageattention
- Repository: https://github.com/woct0rdho/sageattention

## Overview

SageAttention is a GPU-accelerated attention package with standard int8-based attention paths and a separate Blackwell FP4 path. The main workflow accepts query, key and value tensors through the Python API, applies quantization and attention kernels, and returns the output (optionally with LSE). The README also documents variable-length and causal attention and support across several GPU generations. Source excerpts show key calls, but not every dispatch or extension binding, so some implementation links are omitted.

Main entry points: [`sageattention/core.py`](https://github.com/woct0rdho/sageattention/blob/main/sageattention/core.py) (Attention dispatch), [`sageattention3_blackwell/csrc/blackwell/kernel_ws.h`](https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/csrc/blackwell/kernel_ws.h) (Blackwell kernel), [`sageattention3_blackwell/sageattn3/api.py`](https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/sageattn3/api.py) (Blackwell API), [`sageattention/__init__.py`](https://github.com/woct0rdho/sageattention/blob/main/sageattention/__init__.py) (Public API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Python API"]
  node_public_api["Public API<br/>[__init__.py]"]
  node_attention_api["Attention dispatch<br/>[core.py]"]
end

subgraph group_standard["Standard Attention"]
  node_quant_api["Quantization API<br/>[quant.py]"]
  node_quant_block["Block quantization<br/>[quant_per_block.py]"]
  node_quant_varlen["Varlen quantization"]
  node_quant_thread["Per-thread quantization"]
  node_attn_block["Block attention"]
  node_attn_causal["Causal attention"]
  node_attn_varlen["Varlen attention"]
  node_attn_causal_varlen["Causal varlen attention"]
end

subgraph group_cuda["CUDA Kernels"]
  node_fused_cuda["Fused CUDA ops<br/>[fused.h]"]
  node_cuda_bindings["CUDA bindings<br/>[pybind.cpp]"]
  node_qattn_sm80["SM80 attention<br/>[attn_cuda_sm80.h]"]
  node_qattn_sm89["SM89 attention<br/>[attn_cuda_sm89.h]"]
  node_qattn_sm90["SM90 attention<br/>[attn_cuda_sm90.h]"]
end

subgraph group_blackwell["Blackwell FP4"]
  node_blackwell_api["Blackwell API<br/>[api.py]"]
  node_blackwell_kernel["Blackwell kernel<br/>[kernel_ws.h]"]
  node_blackwell_mainloop["TMA mainloop<br/>[mainloop_tma_ws.h]"]
  node_blackwell_epilogue["TMA epilogue<br/>[epilogue_tma_ws.h]"]
  node_blackwell_softmax["Fused softmax<br/>[softmax_fused.h]"]
  node_blackwell_barrier["Pipeline barriers<br/>[named_barrier.h]"]
end

node_caller(("Attention caller"))
node_result["Attention output"]

node_caller -->|"calls"| node_public_api
node_public_api -->|"exports"| node_attention_api
node_attention_api -->|"uses"| node_quant_api
node_attention_api -->|"uses"| node_quant_block
node_attention_api -->|"uses"| node_quant_varlen
node_attention_api -->|"uses"| node_quant_thread
node_attention_api -->|"uses"| node_attn_block
node_attention_api -->|"uses"| node_attn_causal
node_attention_api -->|"uses"| node_attn_varlen
node_attention_api -->|"uses"| node_attn_causal_varlen
node_attention_api -.->|"dispatches"| node_qattn_sm80
node_attention_api -.->|"dispatches"| node_qattn_sm89
node_attention_api -.->|"dispatches"| node_qattn_sm90
node_quant_api -->|"calls"| node_fused_cuda
node_cuda_bindings -->|"binds"| node_fused_cuda
node_attention_api -->|"returns"| node_result
node_caller -.->|"calls"| node_blackwell_api
node_blackwell_api -.->|"invokes"| node_blackwell_kernel
node_blackwell_kernel -->|"uses"| node_blackwell_mainloop
node_blackwell_kernel -->|"uses"| node_blackwell_epilogue
node_blackwell_kernel -->|"uses"| node_blackwell_softmax
node_blackwell_kernel -->|"uses"| node_blackwell_barrier
node_blackwell_api -->|"returns"| node_result

click node_public_api "https://github.com/woct0rdho/sageattention/blob/main/sageattention/__init__.py"
click node_attention_api "https://github.com/woct0rdho/sageattention/blob/main/sageattention/core.py"
click node_quant_api "https://github.com/woct0rdho/sageattention/blob/main/sageattention/quant.py"
click node_quant_block "https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/quant_per_block.py"
click node_quant_varlen "https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/quant_per_block_varlen.py"
click node_quant_thread "https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/quant_per_thread.py"
click node_attn_block "https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/attn_qk_int8_per_block.py"
click node_attn_causal "https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/attn_qk_int8_per_block_causal.py"
click node_attn_varlen "https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/attn_qk_int8_block_varlen.py"
click node_attn_causal_varlen "https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/attn_qk_int8_per_block_causal_varlen.py"
click node_fused_cuda "https://github.com/woct0rdho/sageattention/blob/main/csrc/fused/fused.h"
click node_cuda_bindings "https://github.com/woct0rdho/sageattention/blob/main/csrc/fused/pybind.cpp"
click node_qattn_sm80 "https://github.com/woct0rdho/sageattention/blob/main/csrc/qattn/attn_cuda_sm80.h"
click node_qattn_sm89 "https://github.com/woct0rdho/sageattention/blob/main/csrc/qattn/attn_cuda_sm89.h"
click node_qattn_sm90 "https://github.com/woct0rdho/sageattention/blob/main/csrc/qattn/attn_cuda_sm90.h"
click node_blackwell_api "https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/sageattn3/api.py"
click node_blackwell_kernel "https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/csrc/blackwell/kernel_ws.h"
click node_blackwell_mainloop "https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/csrc/blackwell/mainloop_tma_ws.h"
click node_blackwell_epilogue "https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/csrc/blackwell/epilogue_tma_ws.h"
click node_blackwell_softmax "https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/csrc/blackwell/softmax_fused.h"
click node_blackwell_barrier "https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/csrc/blackwell/named_barrier.h"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_public_api,node_attention_api toneBlue
class node_quant_api,node_quant_block,node_quant_varlen,node_quant_thread,node_attn_block,node_attn_causal,node_attn_varlen,node_attn_causal_varlen toneAmber
class node_fused_cuda,node_cuda_bindings,node_qattn_sm80,node_qattn_sm89,node_qattn_sm90 toneMint
class node_blackwell_api,node_blackwell_kernel,node_blackwell_mainloop,node_blackwell_epilogue,node_blackwell_softmax,node_blackwell_barrier toneRose
class node_caller,node_result toneIndigo
```

## Components

### Python API

- [Public API](https://github.com/woct0rdho/sageattention/blob/main/sageattention/__init__.py): `sageattention/__init__.py`
- [Attention dispatch](https://github.com/woct0rdho/sageattention/blob/main/sageattention/core.py): `sageattention/core.py`

### Standard Attention

- [Quantization API](https://github.com/woct0rdho/sageattention/blob/main/sageattention/quant.py): `sageattention/quant.py`
- [Block quantization](https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/quant_per_block.py): `sageattention/triton/quant_per_block.py`
- [Varlen quantization](https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/quant_per_block_varlen.py): `sageattention/triton/quant_per_block_varlen.py`
- [Per-thread quantization](https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/quant_per_thread.py): `sageattention/triton/quant_per_thread.py`
- [Block attention](https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/attn_qk_int8_per_block.py): `sageattention/triton/attn_qk_int8_per_block.py`
- [Causal attention](https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/attn_qk_int8_per_block_causal.py): `sageattention/triton/attn_qk_int8_per_block_causal.py`
- [Varlen attention](https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/attn_qk_int8_block_varlen.py): `sageattention/triton/attn_qk_int8_block_varlen.py`
- [Causal varlen attention](https://github.com/woct0rdho/sageattention/blob/main/sageattention/triton/attn_qk_int8_per_block_causal_varlen.py): `sageattention/triton/attn_qk_int8_per_block_causal_varlen.py`

### CUDA Kernels

- [Fused CUDA ops](https://github.com/woct0rdho/sageattention/blob/main/csrc/fused/fused.h): `csrc/fused/fused.h`
- [CUDA bindings](https://github.com/woct0rdho/sageattention/blob/main/csrc/fused/pybind.cpp): `csrc/fused/pybind.cpp`
- [SM80 attention](https://github.com/woct0rdho/sageattention/blob/main/csrc/qattn/attn_cuda_sm80.h): `csrc/qattn/attn_cuda_sm80.h`
- [SM89 attention](https://github.com/woct0rdho/sageattention/blob/main/csrc/qattn/attn_cuda_sm89.h): `csrc/qattn/attn_cuda_sm89.h`
- [SM90 attention](https://github.com/woct0rdho/sageattention/blob/main/csrc/qattn/attn_cuda_sm90.h): `csrc/qattn/attn_cuda_sm90.h`

### Blackwell FP4

- [Blackwell API](https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/sageattn3/api.py): `sageattention3_blackwell/sageattn3/api.py`
- [Blackwell kernel](https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/csrc/blackwell/kernel_ws.h): `sageattention3_blackwell/csrc/blackwell/kernel_ws.h`
- [TMA mainloop](https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/csrc/blackwell/mainloop_tma_ws.h): `sageattention3_blackwell/csrc/blackwell/mainloop_tma_ws.h`
- [TMA epilogue](https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/csrc/blackwell/epilogue_tma_ws.h): `sageattention3_blackwell/csrc/blackwell/epilogue_tma_ws.h`
- [Fused softmax](https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/csrc/blackwell/softmax_fused.h): `sageattention3_blackwell/csrc/blackwell/softmax_fused.h`
- [Pipeline barriers](https://github.com/woct0rdho/sageattention/blob/main/sageattention3_blackwell/csrc/blackwell/named_barrier.h): `sageattention3_blackwell/csrc/blackwell/named_barrier.h`

### Other components

- **Attention caller**
- **Attention output**

## Connections

- Attention caller → Public API: calls (evidence: `README.md`)
- Public API → Attention dispatch: exports (evidence: `sageattention/__init__.py`)
- Attention dispatch → Quantization API: uses (evidence: `sageattention/core.py`)
- Attention dispatch → Block quantization: uses (evidence: `sageattention/core.py`)
- Attention dispatch → Varlen quantization: uses (evidence: `sageattention/core.py`)
- Attention dispatch → Per-thread quantization: uses (evidence: `sageattention/core.py`)
- Attention dispatch → Block attention: uses (evidence: `sageattention/core.py`)
- Attention dispatch → Causal attention: uses (evidence: `sageattention/core.py`)
- Attention dispatch → Varlen attention: uses (evidence: `sageattention/core.py`)
- Attention dispatch → Causal varlen attention: uses (evidence: `sageattention/core.py`)
- Attention dispatch → SM80 attention: dispatches
- Attention dispatch → SM89 attention: dispatches
- Attention dispatch → SM90 attention: dispatches
- Quantization API → Fused CUDA ops: calls (evidence: `sageattention/quant.py`)
- CUDA bindings → Fused CUDA ops: binds (evidence: `csrc/fused/pybind.cpp`)
- Attention dispatch → Attention output: returns (evidence: `sageattention/core.py`)
- Attention caller → Blackwell API: calls
- Blackwell API → Blackwell kernel: invokes
- Blackwell kernel → TMA mainloop: uses (evidence: `sageattention3_blackwell/csrc/blackwell/kernel_ws.h`)
- Blackwell kernel → TMA epilogue: uses (evidence: `sageattention3_blackwell/csrc/blackwell/kernel_ws.h`)
- Blackwell kernel → Fused softmax: uses (evidence: `sageattention3_blackwell/csrc/blackwell/kernel_ws.h`)
- Blackwell kernel → Pipeline barriers: uses (evidence: `sageattention3_blackwell/csrc/blackwell/kernel_ws.h`)
- Blackwell API → Attention output: returns (evidence: `sageattention3_blackwell/sageattn3/api.py`)
