How deepseek-ai/flashmla works
FlashMLA is a PyTorch-facing library of hardware-optimized attention kernels for NVIDIA GPUs and Huawei Ascend NPUs. The principal workflow is to pass query, KV-cache, and sparse-index tensors through the Python interface into a native API, which dispatches supported sparse prefill or decoding work to a device kernel and returns attention results. Separate capabilities include fused norm/RoPE/attention/cast, dense prefill and backward attention, and split-KV decode scheduling and combination. The graph reflects documented capabilities and visible source references; some native entry points and kernel implementations were not excerpted, so their links are documented or left unverified.
Main entry points: sparse_decode.cpp Sparse decode API · api.cpp Extension bindings · sparse_prefill.cpp Sparse prefill API · common.h Dispatch and validation
Components 23
Python interface
- Attention interface
flash_mla/flash_mla_interface.py - Fused interface
flash_mla/fused_norm_rope_attn_rope_cast.py
Native API
- Extension bindings
csrc/api/api.cpp - Dispatch and validation
csrc/api/common.h - Sparse prefill API
csrc/api/sparse_prefill.cpp - Sparse decode API
csrc/api/sparse_decode.cpp - Fused kernel API
csrc/api/fused_norm_rope_attn_rope_cast_fwd.cpp - Dense forward API
csrc/api/dense_fwd.cpp - Dense backward API
csrc/api/dense_bwd.cpp
Sparse attention
- CUDA sparse prefill
csrc/cuda_kernels/sm100/prefill/sparse - Ascend sparse kernel
csrc/ascend_kernels/prefill/sparse/kernel_body.h - CUDA sparse decode
csrc/cuda_kernels/sm100/decode/sparse/head64/kernel.h - Decode scheduling
csrc/cuda_kernels/smxx/decode/get_decoding_sched_meta/get_decoding_sched_meta.h - Split-KV combine
csrc/cuda_kernels/smxx/decode/combine/combine.h - Fused CUDA kernels
csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd
Dense attention
- Dense forward kernels
csrc/cuda_kernels/sm100/prefill/dense/kernel/sm100_fmha_fwd_kernel_tma_warpspecialized.hpp - Dense backward kernels
csrc/cuda_kernels/sm100/prefill/dense/device/fmha_device_bwd.hpp - MLA load pipeline
csrc/cuda_kernels/sm100/prefill/dense/common/pipeline_mla.hpp
Runtime support
- KV cache formats
csrc/cuda_kernels/kv_cache_format.h - Device stream access
csrc/device_stream.h - Kernel parameters
csrc/params.h - Kernel utilities
csrc/3rdparty/kerutils/include/kerutils/common/common.h
Other components
- PyTorch caller
Connections 24
- PyTorch caller → Attention interface: calls
README.md - PyTorch caller → Fused interface: calls optionally
README.md - Attention interface → Extension bindings: invokes
- Fused interface → Fused kernel API: invokes
- Extension bindings → Sparse prefill API: routes
- Extension bindings → Sparse decode API: routes
- Extension bindings → Fused kernel API: routes
- Sparse prefill API → CUDA sparse prefill: launches
csrc/api/sparse_prefill.cpp - Sparse prefill API → Ascend sparse kernel: dispatches
csrc/api/sparse_prefill.cpp - Sparse decode API → CUDA sparse decode: launches
csrc/api/sparse_decode.cpp - Sparse decode API → Ascend sparse kernel: dispatches
csrc/api/sparse_decode.cpp - Sparse decode API → Decode scheduling: uses
csrc/api/sparse_decode.cpp - Sparse decode API → Split-KV combine: combines splits
csrc/api/sparse_decode.cpp - Fused kernel API → Fused CUDA kernels: launches
csrc/api/fused_norm_rope_attn_rope_cast_fwd.cpp - Dense forward API → Dense forward kernels: dispatches
README.md - Dense backward API → Dense backward kernels: dispatches
README.md - Dense backward kernels → Dense forward kernels: reuses device layer
csrc/cuda_kernels/sm100/prefill/dense/device/fmha_device_bwd.hpp - Dense forward kernels → MLA load pipeline: uses pipeline
- Dispatch and validation → KV cache formats: dispatches formats
csrc/api/common.h - Sparse decode API → Dispatch and validation: uses dispatch
csrc/api/sparse_decode.cpp - Sparse prefill API → Kernel parameters: sets parameters
csrc/api/sparse_prefill.cpp - Ascend sparse kernel → Kernel parameters: consumes parameters
- Dispatch and validation → Device stream access: uses stream access
csrc/api/common.h - Device stream access → Kernel utilities: uses utilities