# deepseek-ai/flashmla architecture

> Architecture diagram of the GitHub repository deepseek-ai/flashmla, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/deepseek-ai/flashmla
- Repository: https://github.com/deepseek-ai/flashmla

## Overview

FlashMLA is a PyTorch-facing library of hardware-optimized attention kernels for NVIDIA GPUs and Huawei Ascend NPUs. The principal workflow is to pass query, KV-cache, and sparse-index tensors through the Python interface into a native API, which dispatches supported sparse prefill or decoding work to a device kernel and returns attention results. Separate capabilities include fused norm/RoPE/attention/cast, dense prefill and backward attention, and split-KV decode scheduling and combination. The graph reflects documented capabilities and visible source references; some native entry points and kernel implementations were not excerpted, so their links are documented or left unverified.

Main entry points: [`csrc/api/sparse_decode.cpp`](https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/sparse_decode.cpp) (Sparse decode API), [`csrc/api/api.cpp`](https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/api.cpp) (Extension bindings), [`csrc/api/sparse_prefill.cpp`](https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/sparse_prefill.cpp) (Sparse prefill API), [`csrc/api/common.h`](https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/common.h) (Dispatch and validation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Python interface"]
  node_python_api["Attention interface"]
  node_fused_python["Fused interface"]
end

subgraph group_api["Native API"]
  node_native_api["Extension bindings<br/>[api.cpp]"]
  node_api_common["Dispatch and validation<br/>[common.h]"]
  node_sparse_prefill_api["Sparse prefill API<br/>[sparse_prefill.cpp]"]
  node_sparse_decode_api["Sparse decode API<br/>[sparse_decode.cpp]"]
  node_fused_api["Fused kernel API"]
  node_dense_fwd_api["Dense forward API<br/>[dense_fwd.cpp]"]
  node_dense_bwd_api["Dense backward API<br/>[dense_bwd.cpp]"]
end

subgraph group_sparse["Sparse attention"]
  node_cuda_sparse_prefill["CUDA sparse prefill"]
  node_ascend_sparse["Ascend sparse kernel<br/>[kernel_body.h]"]
  node_cuda_sparse_decode["CUDA sparse decode<br/>[kernel.h]"]
  node_decode_schedule["Decode scheduling"]
  node_decode_combine["Split-KV combine<br/>[combine.h]"]
  node_fused_kernels["Fused CUDA kernels"]
end

subgraph group_dense["Dense attention"]
  node_dense_forward["Dense forward kernels"]
  node_dense_backward["Dense backward kernels"]
  node_dense_pipeline["MLA load pipeline<br/>[pipeline_mla.hpp]"]
end

subgraph group_runtime["Runtime support"]
  node_kv_format["KV cache formats<br/>[kv_cache_format.h]"]
  node_device_stream["Device stream access<br/>[device_stream.h]"]
  node_params["Kernel parameters<br/>[params.h]"]
  node_kerutils["Kernel utilities<br/>[common.h]"]
end

node_caller(("PyTorch caller"))

node_caller -->|"calls"| node_python_api
node_caller -.->|"calls optionally"| node_fused_python
node_python_api -.->|"invokes"| node_native_api
node_fused_python -.->|"invokes"| node_fused_api
node_native_api -.->|"routes"| node_sparse_prefill_api
node_native_api -.->|"routes"| node_sparse_decode_api
node_native_api -.->|"routes"| node_fused_api
node_sparse_prefill_api -->|"launches"| node_cuda_sparse_prefill
node_sparse_prefill_api -->|"dispatches"| node_ascend_sparse
node_sparse_decode_api -->|"launches"| node_cuda_sparse_decode
node_sparse_decode_api -->|"dispatches"| node_ascend_sparse
node_sparse_decode_api -->|"uses"| node_decode_schedule
node_sparse_decode_api -->|"combines splits"| node_decode_combine
node_fused_api -->|"launches"| node_fused_kernels
node_dense_fwd_api -.->|"dispatches"| node_dense_forward
node_dense_bwd_api -.->|"dispatches"| node_dense_backward
node_dense_backward -.->|"reuses device layer"| node_dense_forward
node_dense_forward -.->|"uses pipeline"| node_dense_pipeline
node_api_common -->|"dispatches formats"| node_kv_format
node_sparse_decode_api -->|"uses dispatch"| node_api_common
node_sparse_prefill_api -->|"sets parameters"| node_params
node_ascend_sparse -->|"consumes parameters"| node_params
node_api_common -->|"uses stream access"| node_device_stream
node_device_stream -.->|"uses utilities"| node_kerutils

click node_python_api "https://github.com/deepseek-ai/flashmla/blob/main/flash_mla/flash_mla_interface.py"
click node_fused_python "https://github.com/deepseek-ai/flashmla/blob/main/flash_mla/fused_norm_rope_attn_rope_cast.py"
click node_native_api "https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/api.cpp"
click node_api_common "https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/common.h"
click node_sparse_prefill_api "https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/sparse_prefill.cpp"
click node_sparse_decode_api "https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/sparse_decode.cpp"
click node_fused_api "https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/fused_norm_rope_attn_rope_cast_fwd.cpp"
click node_dense_fwd_api "https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/dense_fwd.cpp"
click node_dense_bwd_api "https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/dense_bwd.cpp"
click node_cuda_sparse_prefill "https://github.com/deepseek-ai/flashmla/tree/main/csrc/cuda_kernels/sm100/prefill/sparse"
click node_ascend_sparse "https://github.com/deepseek-ai/flashmla/blob/main/csrc/ascend_kernels/prefill/sparse/kernel_body.h"
click node_cuda_sparse_decode "https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/sm100/decode/sparse/head64/kernel.h"
click node_decode_schedule "https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/smxx/decode/get_decoding_sched_meta/get_decoding_sched_meta.h"
click node_decode_combine "https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/smxx/decode/combine/combine.h"
click node_fused_kernels "https://github.com/deepseek-ai/flashmla/tree/main/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd"
click node_dense_forward "https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/sm100/prefill/dense/kernel/sm100_fmha_fwd_kernel_tma_warpspecialized.hpp"
click node_dense_backward "https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/sm100/prefill/dense/device/fmha_device_bwd.hpp"
click node_dense_pipeline "https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/sm100/prefill/dense/common/pipeline_mla.hpp"
click node_kv_format "https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/kv_cache_format.h"
click node_device_stream "https://github.com/deepseek-ai/flashmla/blob/main/csrc/device_stream.h"
click node_params "https://github.com/deepseek-ai/flashmla/blob/main/csrc/params.h"
click node_kerutils "https://github.com/deepseek-ai/flashmla/blob/main/csrc/3rdparty/kerutils/include/kerutils/common/common.h"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_python_api,node_fused_python toneBlue
class node_native_api,node_api_common,node_sparse_prefill_api,node_sparse_decode_api,node_fused_api,node_dense_fwd_api,node_dense_bwd_api toneAmber
class node_cuda_sparse_prefill,node_ascend_sparse,node_cuda_sparse_decode,node_decode_schedule,node_decode_combine,node_fused_kernels toneMint
class node_dense_forward,node_dense_backward,node_dense_pipeline toneRose
class node_kv_format,node_device_stream,node_params,node_kerutils,node_caller toneIndigo
```

## Components

### Python interface

- [Attention interface](https://github.com/deepseek-ai/flashmla/blob/main/flash_mla/flash_mla_interface.py): `flash_mla/flash_mla_interface.py`
- [Fused interface](https://github.com/deepseek-ai/flashmla/blob/main/flash_mla/fused_norm_rope_attn_rope_cast.py): `flash_mla/fused_norm_rope_attn_rope_cast.py`

### Native API

- [Extension bindings](https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/api.cpp): `csrc/api/api.cpp`
- [Dispatch and validation](https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/common.h): `csrc/api/common.h`
- [Sparse prefill API](https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/sparse_prefill.cpp): `csrc/api/sparse_prefill.cpp`
- [Sparse decode API](https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/sparse_decode.cpp): `csrc/api/sparse_decode.cpp`
- [Fused kernel API](https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/fused_norm_rope_attn_rope_cast_fwd.cpp): `csrc/api/fused_norm_rope_attn_rope_cast_fwd.cpp`
- [Dense forward API](https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/dense_fwd.cpp): `csrc/api/dense_fwd.cpp`
- [Dense backward API](https://github.com/deepseek-ai/flashmla/blob/main/csrc/api/dense_bwd.cpp): `csrc/api/dense_bwd.cpp`

### Sparse attention

- [CUDA sparse prefill](https://github.com/deepseek-ai/flashmla/tree/main/csrc/cuda_kernels/sm100/prefill/sparse): `csrc/cuda_kernels/sm100/prefill/sparse`
- [Ascend sparse kernel](https://github.com/deepseek-ai/flashmla/blob/main/csrc/ascend_kernels/prefill/sparse/kernel_body.h): `csrc/ascend_kernels/prefill/sparse/kernel_body.h`
- [CUDA sparse decode](https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/sm100/decode/sparse/head64/kernel.h): `csrc/cuda_kernels/sm100/decode/sparse/head64/kernel.h`
- [Decode scheduling](https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/smxx/decode/get_decoding_sched_meta/get_decoding_sched_meta.h): `csrc/cuda_kernels/smxx/decode/get_decoding_sched_meta/get_decoding_sched_meta.h`
- [Split-KV combine](https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/smxx/decode/combine/combine.h): `csrc/cuda_kernels/smxx/decode/combine/combine.h`
- [Fused CUDA kernels](https://github.com/deepseek-ai/flashmla/tree/main/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd): `csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd`

### Dense attention

- [Dense forward kernels](https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/sm100/prefill/dense/kernel/sm100_fmha_fwd_kernel_tma_warpspecialized.hpp): `csrc/cuda_kernels/sm100/prefill/dense/kernel/sm100_fmha_fwd_kernel_tma_warpspecialized.hpp`
- [Dense backward kernels](https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/sm100/prefill/dense/device/fmha_device_bwd.hpp): `csrc/cuda_kernels/sm100/prefill/dense/device/fmha_device_bwd.hpp`
- [MLA load pipeline](https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/sm100/prefill/dense/common/pipeline_mla.hpp): `csrc/cuda_kernels/sm100/prefill/dense/common/pipeline_mla.hpp`

### Runtime support

- [KV cache formats](https://github.com/deepseek-ai/flashmla/blob/main/csrc/cuda_kernels/kv_cache_format.h): `csrc/cuda_kernels/kv_cache_format.h`
- [Device stream access](https://github.com/deepseek-ai/flashmla/blob/main/csrc/device_stream.h): `csrc/device_stream.h`
- [Kernel parameters](https://github.com/deepseek-ai/flashmla/blob/main/csrc/params.h): `csrc/params.h`
- [Kernel utilities](https://github.com/deepseek-ai/flashmla/blob/main/csrc/3rdparty/kerutils/include/kerutils/common/common.h): `csrc/3rdparty/kerutils/include/kerutils/common/common.h`

### Other components

- **PyTorch caller**

## Connections

- PyTorch caller → Attention interface: calls (evidence: `README.md`)
- PyTorch caller → Fused interface: calls optionally (evidence: `README.md`)
- Attention interface → Extension bindings: invokes
- Fused interface → Fused kernel API: invokes
- Extension bindings → Sparse prefill API: routes
- Extension bindings → Sparse decode API: routes
- Extension bindings → Fused kernel API: routes
- Sparse prefill API → CUDA sparse prefill: launches (evidence: `csrc/api/sparse_prefill.cpp`)
- Sparse prefill API → Ascend sparse kernel: dispatches (evidence: `csrc/api/sparse_prefill.cpp`)
- Sparse decode API → CUDA sparse decode: launches (evidence: `csrc/api/sparse_decode.cpp`)
- Sparse decode API → Ascend sparse kernel: dispatches (evidence: `csrc/api/sparse_decode.cpp`)
- Sparse decode API → Decode scheduling: uses (evidence: `csrc/api/sparse_decode.cpp`)
- Sparse decode API → Split-KV combine: combines splits (evidence: `csrc/api/sparse_decode.cpp`)
- Fused kernel API → Fused CUDA kernels: launches (evidence: `csrc/api/fused_norm_rope_attn_rope_cast_fwd.cpp`)
- Dense forward API → Dense forward kernels: dispatches (evidence: `README.md`)
- Dense backward API → Dense backward kernels: dispatches (evidence: `README.md`)
- Dense backward kernels → Dense forward kernels: reuses device layer (evidence: `csrc/cuda_kernels/sm100/prefill/dense/device/fmha_device_bwd.hpp`)
- Dense forward kernels → MLA load pipeline: uses pipeline
- Dispatch and validation → KV cache formats: dispatches formats (evidence: `csrc/api/common.h`)
- Sparse decode API → Dispatch and validation: uses dispatch (evidence: `csrc/api/sparse_decode.cpp`)
- Sparse prefill API → Kernel parameters: sets parameters (evidence: `csrc/api/sparse_prefill.cpp`)
- Ascend sparse kernel → Kernel parameters: consumes parameters
- Dispatch and validation → Device stream access: uses stream access (evidence: `csrc/api/common.h`)
- Device stream access → Kernel utilities: uses utilities
