How kvcache-ai/mooncake works
Purpose and Product Surface
Mooncake is C++-centric infrastructure for disaggregated LLM inference/training: it moves tensors/KV cache efficiently, manages a distributed KV-cache and weight pool, and supports elastic MoE communication. The top-level build composes the major native subsystems in CMakeLists.txt; packaged Python entry points and CLI live in mooncake-wheel/mooncake/cli.py and mooncake-wheel/mooncake/__init__.py.
Transfer Engine: Data Plane
The Transfer Engine is the core asynchronous data-movement layer. Callers register memory and submit batched transfers; the engine identifies source/destination memory and topology, then selects and executes a suitable transport path. Public C/C++ boundaries are mooncake-transfer-engine/include/transfer_engine.h, mooncake-transfer-engine/include/transfer_engine_c.h, and implementation orchestration is in mooncake-transfer-engine/src/transfer_engine_impl.cpp.
It supports host/device memory and heterogeneous fabrics, including TCP, RDMA, EFA, NVLink, NCCL, NVMe-oF, CXL, HIP, and Ascend-family transports. Concrete transport boundaries include mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp, mooncake-transfer-engine/src/transport/tcp_transport/tcp_transport.cpp, and mooncake-transfer-engine/src/transport/nvmeof_transport/nvmeof_transport.cpp.
TENT Runtime
TENT is a newer transfer runtime within the Transfer Engine, with explicit topology, admission, QoS, receiver-credit, progress-worker, and transport-selection stages. It separates control/metadata coordination from transfer execution and dynamically loads/selects transports. Core paths are mooncake-transfer-engine/tent/src/runtime/transfer_engine_impl.cpp, mooncake-transfer-engine/tent/src/runtime/transport_selector.cpp, and mooncake-transfer-engine/tent/src/runtime/control_plane.cpp.
TENT metadata backends support etcd, HTTP, and Redis (mooncake-transfer-engine/tent/src/metastore/etcd.cpp, mooncake-transfer-engine/tent/src/metastore/http.cpp, mooncake-transfer-engine/tent/src/metastore/redis.cpp), making external metadata service availability a runtime boundary.
Mooncake Store: Distributed Object/KV Cache
Mooncake Store builds distributed object storage on Transfer Engine. A master service owns metadata, placement, replica lifecycle, allocation, eviction, and client coordination; clients perform high-bandwidth data transfer to segments. Main service/client entry implementations are mooncake-store/src/master.cpp, mooncake-store/src/master_service.cpp, and mooncake-store/src/real_client.cpp.
The central flow is: application puts/gets an object → master resolves metadata and replicas → client allocates/registers buffers and schedules Transfer Engine tasks → data moves directly to/from memory or backing storage. Key boundaries are mooncake-store/src/metadata_store.cpp, mooncake-store/src/transfer_task.cpp, and mooncake-store/src/segment.cpp.
Storage Tiers, Devices, and Persistence
Store supports DRAM/GPU-accessible memory plus SSD/NVMe and distributed filesystem backends, enabling promotion/offload and cache hierarchy. Device abstraction and CUDA/HIP/Ascend implementations are in mooncake-store/src/device/accelerator_registry.cpp, mooncake-store/src/device/cuda_ipc_buffer.cpp, and mooncake-store/src/device/ascend_accelerator_device.cpp. Storage boundaries include mooncake-store/src/file_storage.cpp, mooncake-store/src/spdk/spdk_wrapper.cpp, and mooncake-store/src/storage/distributed/distributed_storage_backend.cpp.
Availability and Cluster Control
Store has optional HA around the master: leadership can use etcd, Kubernetes leases, or Redis; state continuity uses ordered oplog replication and snapshots to local files or S3. Core control paths are mooncake-store/src/ha/leadership/master_service_supervisor.cpp, mooncake-store/src/ha/oplog/ordered_oplog_writer.cpp, and mooncake-store/src/ha/standby_controller.cpp. External infrastructure therefore includes etcd, Redis, Kubernetes API/leases, S3-compatible object storage, SPDK/NVMe, and Prometheus/Grafana (monitoring/docker-compose.yml).
Elastic MoE and PyTorch Integration
Mooncake EP provides CUDA expert-parallel dispatch/combine kernels with elastic rank awareness; Mooncake PG provides collective/P2P communication plus agent/coordinator control-plane recovery. EP’s native/Python boundary is mooncake-ep/src/ep_py.cpp and mooncake-ep/src/mooncake_ep_elastic_kernel.cu. PG’s runtime is centered on mooncake-pg/src/mooncake_communicator.cpp, mooncake-pg/src/control_plane/coordinator.cpp, and its PyTorch backend binding mooncake-pg/torch/src/mooncake_backend.cpp.
Python integrations expose Store, Transfer Engine, EP, and PG through pybind-backed modules in mooncake-integration/store/store_py.cpp and mooncake-integration/transfer_engine/transfer_engine_py.cpp, then package them for inference ecosystems such as vLLM and SGLang.
Main entry points: transfer_engine_py.cpp PyTorch bindings · master_service.cpp Store master service · transfer_engine.h Transfer Engine API · CMakeLists.txt Native build
Components 21
Transfer Engine
Asynchronous tensor and memory movement.
- Transfer Engine API · Memory registration and batched transfer submission boundary.
mooncake-transfer-engine/include/transfer_engine.h - Transfer orchestration · Identifies memory/topology and schedules transport work.
mooncake-transfer-engine/src/transfer_engine_impl.cpp - Transport fabrics · TCP, RDMA, NVMe-oF and accelerator/network transports.
mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp - TENT runtime · Applies topology, admission, QoS, credits and progress stages.
mooncake-transfer-engine/tent/src/runtime/transfer_engine_impl.cpp - TENT transport selector · Loads and selects execution transports.
mooncake-transfer-engine/tent/src/runtime/transport_selector.cpp - TENT control plane · Separates transfer coordination from execution.
mooncake-transfer-engine/tent/src/runtime/control_plane.cpp - Metadata services · Runtime metadata backend availability boundary.
mooncake-transfer-engine/tent/src/metastore/etcd.cpp
Mooncake Store
Distributed object and KV-cache service.
- Store client · Allocates buffers and creates transfer tasks.
mooncake-store/src/real_client.cpp - Store master service · Coordinates placement, replicas, allocation and eviction.
mooncake-store/src/master_service.cpp - Store metadata · Maintains object and replica state.
mooncake-store/src/metadata_store.cpp - Store transfer tasks · Bridges client data operations to Transfer Engine.
mooncake-store/src/transfer_task.cpp - Storage segments · Segment-level object data boundary.
mooncake-store/src/segment.cpp - Storage tiers · Supports cache promotion, offload and backing storage.
mooncake-store/src/file_storage.cpp
Cluster Control
Optional master availability and persistence.
- Master HA supervisor · Uses etcd, Kubernetes leases, or Redis leadership.
mooncake-store/src/ha/leadership/master_service_supervisor.cpp - Ordered oplog · Replicates master state for standby continuity.
mooncake-store/src/ha/oplog/ordered_oplog_writer.cpp - Snapshot storage · Persists recovery snapshots.
mooncake-store/src/ha/standby_controller.cpp
Elastic MoE
PyTorch-facing expert-parallel communication.
- Mooncake EP · CUDA dispatch/combine kernels with elastic rank awareness.
mooncake-ep/src/mooncake_ep_elastic_kernel.cu - Mooncake PG · Provides communicator recovery and coordinator control.
mooncake-pg/src/mooncake_communicator.cpp - PyTorch bindings · Exposes Store, Transfer Engine, EP and PG to Python.
mooncake-integration/transfer_engine/transfer_engine_py.cpp
Other components
- Native build · Composes the major native subsystems.
CMakeLists.txt - Python package and CLI · Packages native integrations for inference users.
mooncake-wheel/mooncake/cli.py
Connections 24
- Native build → Transfer Engine API: builds
- Native build → Store master service: builds
- Native build → Mooncake EP: builds
- Transfer Engine API → Transfer orchestration: submits transfers
- Transfer orchestration → Transport fabrics: selects execution path
- TENT runtime → TENT transport selector: selects transport
- TENT runtime → TENT control plane: coordinates metadata
- TENT transport selector → Transport fabrics: loads/uses
- TENT control plane → Metadata services: uses
- Store client → Store master service: resolves objects and replicas
- Store master service → Store metadata: owns
- Store client → Store transfer tasks: schedules
- Store transfer tasks → Transfer Engine API: moves data through
- Store transfer tasks → Storage segments: reads/writes
- Storage segments → Storage tiers: backs onto
- Master HA supervisor → Store master service: supervises leader
- Store master service → Ordered oplog: replicates state
- Ordered oplog → Snapshot storage: snapshots to
- Mooncake EP → Mooncake PG: uses elastic communication
- PyTorch bindings → Store client: exposes
- PyTorch bindings → Transfer Engine API: exposes
- PyTorch bindings → Mooncake EP: exposes
- PyTorch bindings → Mooncake PG: exposes
- Python package and CLI → PyTorch bindings: packages