# kvcache-ai/mooncake architecture

> Architecture diagram of the GitHub repository kvcache-ai/mooncake, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kvcache-ai/mooncake
- Repository: https://github.com/kvcache-ai/mooncake

## Overview

#### Purpose and Product Surface
Mooncake is C++-centric infrastructure for disaggregated LLM inference/training: it moves tensors/KV cache efficiently, manages a distributed KV-cache and weight pool, and supports elastic MoE communication. The top-level build composes the major native subsystems in `CMakeLists.txt`; packaged Python entry points and CLI live in `mooncake-wheel/mooncake/cli.py` and `mooncake-wheel/mooncake/__init__.py`.

#### Transfer Engine: Data Plane
The Transfer Engine is the core asynchronous data-movement layer. Callers register memory and submit batched transfers; the engine identifies source/destination memory and topology, then selects and executes a suitable transport path. Public C/C++ boundaries are `mooncake-transfer-engine/include/transfer_engine.h`, `mooncake-transfer-engine/include/transfer_engine_c.h`, and implementation orchestration is in `mooncake-transfer-engine/src/transfer_engine_impl.cpp`.

It supports host/device memory and heterogeneous fabrics, including TCP, RDMA, EFA, NVLink, NCCL, NVMe-oF, CXL, HIP, and Ascend-family transports. Concrete transport boundaries include `mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp`, `mooncake-transfer-engine/src/transport/tcp_transport/tcp_transport.cpp`, and `mooncake-transfer-engine/src/transport/nvmeof_transport/nvmeof_transport.cpp`.

#### TENT Runtime
TENT is a newer transfer runtime within the Transfer Engine, with explicit topology, admission, QoS, receiver-credit, progress-worker, and transport-selection stages. It separates control/metadata coordination from transfer execution and dynamically loads/selects transports. Core paths are `mooncake-transfer-engine/tent/src/runtime/transfer_engine_impl.cpp`, `mooncake-transfer-engine/tent/src/runtime/transport_selector.cpp`, and `mooncake-transfer-engine/tent/src/runtime/control_plane.cpp`.

TENT metadata backends support etcd, HTTP, and Redis (`mooncake-transfer-engine/tent/src/metastore/etcd.cpp`, `mooncake-transfer-engine/tent/src/metastore/http.cpp`, `mooncake-transfer-engine/tent/src/metastore/redis.cpp`), making external metadata service availability a runtime boundary.

#### Mooncake Store: Distributed Object/KV Cache
Mooncake Store builds distributed object storage on Transfer Engine. A master service owns metadata, placement, replica lifecycle, allocation, eviction, and client coordination; clients perform high-bandwidth data transfer to segments. Main service/client entry implementations are `mooncake-store/src/master.cpp`, `mooncake-store/src/master_service.cpp`, and `mooncake-store/src/real_client.cpp`.

The central flow is: application puts/gets an object → master resolves metadata and replicas → client allocates/registers buffers and schedules Transfer Engine tasks → data moves directly to/from memory or backing storage. Key boundaries are `mooncake-store/src/metadata_store.cpp`, `mooncake-store/src/transfer_task.cpp`, and `mooncake-store/src/segment.cpp`.

#### Storage Tiers, Devices, and Persistence
Store supports DRAM/GPU-accessible memory plus SSD/NVMe and distributed filesystem backends, enabling promotion/offload and cache hierarchy. Device abstraction and CUDA/HIP/Ascend implementations are in `mooncake-store/src/device/accelerator_registry.cpp`, `mooncake-store/src/device/cuda_ipc_buffer.cpp`, and `mooncake-store/src/device/ascend_accelerator_device.cpp`. Storage boundaries include `mooncake-store/src/file_storage.cpp`, `mooncake-store/src/spdk/spdk_wrapper.cpp`, and `mooncake-store/src/storage/distributed/distributed_storage_backend.cpp`.

#### Availability and Cluster Control
Store has optional HA around the master: leadership can use etcd, Kubernetes leases, or Redis; state continuity uses ordered oplog replication and snapshots to local files or S3. Core control paths are `mooncake-store/src/ha/leadership/master_service_supervisor.cpp`, `mooncake-store/src/ha/oplog/ordered_oplog_writer.cpp`, and `mooncake-store/src/ha/standby_controller.cpp`. External infrastructure therefore includes etcd, Redis, Kubernetes API/leases, S3-compatible object storage, SPDK/NVMe, and Prometheus/Grafana (`monitoring/docker-compose.yml`).

#### Elastic MoE and PyTorch Integration
Mooncake EP provides CUDA expert-parallel dispatch/combine kernels with elastic rank awareness; Mooncake PG provides collective/P2P communication plus agent/coordinator control-plane recovery. EP’s native/Python boundary is `mooncake-ep/src/ep_py.cpp` and `mooncake-ep/src/mooncake_ep_elastic_kernel.cu`. PG’s runtime is centered on `mooncake-pg/src/mooncake_communicator.cpp`, `mooncake-pg/src/control_plane/coordinator.cpp`, and its PyTorch backend binding `mooncake-pg/torch/src/mooncake_backend.cpp`.

Python integrations expose Store, Transfer Engine, EP, and PG through pybind-backed modules in `mooncake-integration/store/store_py.cpp` and `mooncake-integration/transfer_engine/transfer_engine_py.cpp`, then package them for inference ecosystems such as vLLM and SGLang.

Main entry points: [`mooncake-integration/transfer_engine/transfer_engine_py.cpp`](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-integration/transfer_engine/transfer_engine_py.cpp) (PyTorch bindings), [`mooncake-store/src/master_service.cpp`](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/master_service.cpp) (Store master service), [`mooncake-transfer-engine/include/transfer_engine.h`](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/include/transfer_engine.h) (Transfer Engine API), [`CMakeLists.txt`](https://github.com/kvcache-ai/mooncake/blob/main/CMakeLists.txt) (Native build)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_transfer["Transfer Engine"]
  node_transfer_api["Transfer Engine API<br/>C/C++ API<br/>[transfer_engine.h]"]
  node_transfer_orchestrator["Transfer orchestration<br/>async data plane"]
  node_transport_fabrics["Transport fabrics<br/>transport implementations<br/>[rdma_transport.cpp]"]
  node_tent_runtime["TENT runtime<br/>transfer runtime"]
  node_tent_selector["TENT transport selector<br/>dynamic transport selection"]
  node_tent_control["TENT control plane<br/>metadata coordination<br/>[control_plane.cpp]"]
  node_metadata_services[("Metadata services<br/>external etcd/HTTP/Redis<br/>[etcd.cpp]")]
end

subgraph group_store["Mooncake Store"]
  node_store_client["Store client<br/>object cache client<br/>[real_client.cpp]"]
  node_store_master["Store master service<br/>metadata and placement service<br/>[master_service.cpp]"]
  node_store_metadata[("Store metadata<br/>object metadata<br/>[metadata_store.cpp]")]
  node_transfer_tasks["Store transfer tasks<br/>transfer scheduling<br/>[transfer_task.cpp]"]
  node_segments["Storage segments<br/>data placement<br/>[segment.cpp]"]
  node_storage_tiers[("Storage tiers<br/>DRAM/GPU/SSD/distributed FS<br/>[file_storage.cpp]")]
end

subgraph group_ha["Cluster Control"]
  node_master_ha["Master HA supervisor<br/>leadership control"]
  node_oplog["Ordered oplog<br/>state replication"]
  node_snapshot_storage[("Snapshot storage<br/>local files or S3")]
end

subgraph group_moe["Elastic MoE"]
  node_ep_runtime["Mooncake EP<br/>elastic expert parallelism"]
  node_pg_runtime["Mooncake PG<br/>collective and P2P runtime"]
  node_pytorch_bindings["PyTorch bindings<br/>pybind backend modules"]
end

node_native_build["Native build<br/>CMake composition<br/>[CMakeLists.txt]"]
node_python_package["Python package and CLI<br/>Python entry points<br/>[cli.py]"]

node_native_build -->|"builds"| node_transfer_api
node_native_build -->|"builds"| node_store_master
node_native_build -->|"builds"| node_ep_runtime
node_transfer_api -->|"submits transfers"| node_transfer_orchestrator
node_transfer_orchestrator -->|"selects execution path"| node_transport_fabrics
node_tent_runtime -->|"selects transport"| node_tent_selector
node_tent_runtime -->|"coordinates metadata"| node_tent_control
node_tent_selector -->|"loads/uses"| node_transport_fabrics
node_tent_control -->|"uses"| node_metadata_services
node_store_client -->|"resolves objects and replicas"| node_store_master
node_store_master -->|"owns"| node_store_metadata
node_store_client -->|"schedules"| node_transfer_tasks
node_transfer_tasks -->|"moves data through"| node_transfer_api
node_transfer_tasks -->|"reads/writes"| node_segments
node_segments -->|"backs onto"| node_storage_tiers
node_master_ha -->|"supervises leader"| node_store_master
node_store_master -->|"replicates state"| node_oplog
node_oplog -->|"snapshots to"| node_snapshot_storage
node_ep_runtime -->|"uses elastic communication"| node_pg_runtime
node_pytorch_bindings -->|"exposes"| node_store_client
node_pytorch_bindings -->|"exposes"| node_transfer_api
node_pytorch_bindings -->|"exposes"| node_ep_runtime
node_pytorch_bindings -->|"exposes"| node_pg_runtime
node_python_package -->|"packages"| node_pytorch_bindings

click node_native_build "https://github.com/kvcache-ai/mooncake/blob/main/CMakeLists.txt"
click node_python_package "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-wheel/mooncake/cli.py"
click node_transfer_api "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/include/transfer_engine.h"
click node_transfer_orchestrator "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/src/transfer_engine_impl.cpp"
click node_transport_fabrics "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp"
click node_tent_runtime "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/tent/src/runtime/transfer_engine_impl.cpp"
click node_tent_selector "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/tent/src/runtime/transport_selector.cpp"
click node_tent_control "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/tent/src/runtime/control_plane.cpp"
click node_metadata_services "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/tent/src/metastore/etcd.cpp"
click node_store_client "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/real_client.cpp"
click node_store_master "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/master_service.cpp"
click node_store_metadata "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/metadata_store.cpp"
click node_transfer_tasks "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/transfer_task.cpp"
click node_segments "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/segment.cpp"
click node_storage_tiers "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/file_storage.cpp"
click node_master_ha "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/ha/leadership/master_service_supervisor.cpp"
click node_oplog "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/ha/oplog/ordered_oplog_writer.cpp"
click node_snapshot_storage "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/ha/standby_controller.cpp"
click node_ep_runtime "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-ep/src/mooncake_ep_elastic_kernel.cu"
click node_pg_runtime "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-pg/src/mooncake_communicator.cpp"
click node_pytorch_bindings "https://github.com/kvcache-ai/mooncake/blob/main/mooncake-integration/transfer_engine/transfer_engine_py.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_transfer_api,node_transfer_orchestrator,node_transport_fabrics,node_tent_runtime,node_tent_selector,node_tent_control,node_metadata_services toneBlue
class node_store_client,node_store_master,node_store_metadata,node_transfer_tasks,node_segments,node_storage_tiers toneAmber
class node_master_ha,node_oplog,node_snapshot_storage toneMint
class node_ep_runtime,node_pg_runtime,node_pytorch_bindings toneRose
class node_native_build,node_python_package toneNeutral
```

## Components

### Transfer Engine

Asynchronous tensor and memory movement.

- [Transfer Engine API](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/include/transfer_engine.h): C/C++ API · Memory registration and batched transfer submission boundary. · `mooncake-transfer-engine/include/transfer_engine.h`
- [Transfer orchestration](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/src/transfer_engine_impl.cpp): async data plane · Identifies memory/topology and schedules transport work. · `mooncake-transfer-engine/src/transfer_engine_impl.cpp`
- [Transport fabrics](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp): transport implementations · TCP, RDMA, NVMe-oF and accelerator/network transports. · `mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp`
- [TENT runtime](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/tent/src/runtime/transfer_engine_impl.cpp): transfer runtime · Applies topology, admission, QoS, credits and progress stages. · `mooncake-transfer-engine/tent/src/runtime/transfer_engine_impl.cpp`
- [TENT transport selector](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/tent/src/runtime/transport_selector.cpp): dynamic transport selection · Loads and selects execution transports. · `mooncake-transfer-engine/tent/src/runtime/transport_selector.cpp`
- [TENT control plane](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/tent/src/runtime/control_plane.cpp): metadata coordination · Separates transfer coordination from execution. · `mooncake-transfer-engine/tent/src/runtime/control_plane.cpp`
- [Metadata services](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/tent/src/metastore/etcd.cpp): external etcd/HTTP/Redis · Runtime metadata backend availability boundary. · `mooncake-transfer-engine/tent/src/metastore/etcd.cpp`

### Mooncake Store

Distributed object and KV-cache service.

- [Store client](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/real_client.cpp): object cache client · Allocates buffers and creates transfer tasks. · `mooncake-store/src/real_client.cpp`
- [Store master service](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/master_service.cpp): metadata and placement service · Coordinates placement, replicas, allocation and eviction. · `mooncake-store/src/master_service.cpp`
- [Store metadata](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/metadata_store.cpp): object metadata · Maintains object and replica state. · `mooncake-store/src/metadata_store.cpp`
- [Store transfer tasks](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/transfer_task.cpp): transfer scheduling · Bridges client data operations to Transfer Engine. · `mooncake-store/src/transfer_task.cpp`
- [Storage segments](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/segment.cpp): data placement · Segment-level object data boundary. · `mooncake-store/src/segment.cpp`
- [Storage tiers](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/file_storage.cpp): DRAM/GPU/SSD/distributed FS · Supports cache promotion, offload and backing storage. · `mooncake-store/src/file_storage.cpp`

### Cluster Control

Optional master availability and persistence.

- [Master HA supervisor](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/ha/leadership/master_service_supervisor.cpp): leadership control · Uses etcd, Kubernetes leases, or Redis leadership. · `mooncake-store/src/ha/leadership/master_service_supervisor.cpp`
- [Ordered oplog](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/ha/oplog/ordered_oplog_writer.cpp): state replication · Replicates master state for standby continuity. · `mooncake-store/src/ha/oplog/ordered_oplog_writer.cpp`
- [Snapshot storage](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/src/ha/standby_controller.cpp): local files or S3 · Persists recovery snapshots. · `mooncake-store/src/ha/standby_controller.cpp`

### Elastic MoE

PyTorch-facing expert-parallel communication.

- [Mooncake EP](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-ep/src/mooncake_ep_elastic_kernel.cu): elastic expert parallelism · CUDA dispatch/combine kernels with elastic rank awareness. · `mooncake-ep/src/mooncake_ep_elastic_kernel.cu`
- [Mooncake PG](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-pg/src/mooncake_communicator.cpp): collective and P2P runtime · Provides communicator recovery and coordinator control. · `mooncake-pg/src/mooncake_communicator.cpp`
- [PyTorch bindings](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-integration/transfer_engine/transfer_engine_py.cpp): pybind backend modules · Exposes Store, Transfer Engine, EP and PG to Python. · `mooncake-integration/transfer_engine/transfer_engine_py.cpp`

### Other components

- [Native build](https://github.com/kvcache-ai/mooncake/blob/main/CMakeLists.txt): CMake composition · Composes the major native subsystems. · `CMakeLists.txt`
- [Python package and CLI](https://github.com/kvcache-ai/mooncake/blob/main/mooncake-wheel/mooncake/cli.py): Python entry points · Packages native integrations for inference users. · `mooncake-wheel/mooncake/cli.py`

## Connections

- Native build → Transfer Engine API: builds
- Native build → Store master service: builds
- Native build → Mooncake EP: builds
- Transfer Engine API → Transfer orchestration: submits transfers
- Transfer orchestration → Transport fabrics: selects execution path
- TENT runtime → TENT transport selector: selects transport
- TENT runtime → TENT control plane: coordinates metadata
- TENT transport selector → Transport fabrics: loads/uses
- TENT control plane → Metadata services: uses
- Store client → Store master service: resolves objects and replicas
- Store master service → Store metadata: owns
- Store client → Store transfer tasks: schedules
- Store transfer tasks → Transfer Engine API: moves data through
- Store transfer tasks → Storage segments: reads/writes
- Storage segments → Storage tiers: backs onto
- Master HA supervisor → Store master service: supervises leader
- Store master service → Ordered oplog: replicates state
- Ordered oplog → Snapshot storage: snapshots to
- Mooncake EP → Mooncake PG: uses elastic communication
- PyTorch bindings → Store client: exposes
- PyTorch bindings → Transfer Engine API: exposes
- PyTorch bindings → Mooncake EP: exposes
- PyTorch bindings → Mooncake PG: exposes
- Python package and CLI → PyTorch bindings: packages
