# tencent/ksanallm architecture

> Architecture diagram of the GitHub repository tencent/ksanallm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent/ksanallm
- Repository: https://github.com/tencent/ksanallm

## Overview

KsanaLLM is an LLM inference and serving engine. A request enters through the Python engine, which prepares input and runs plugin preprocessing before invoking serving; the C++ service delegates handling to an endpoint and the inference engine manages runtime execution. The documented product also includes streaming and multiple decoding methods, prefix caching, multi-GPU execution, and GPU/NPU kernels. Source excerpts show service, engine, cache, and transfer responsibilities; several capability-specific connections are based on the README and tree, since their implementation was not sampled.

Main entry points: [`csrc/engine/inference_engine.cpp`](https://github.com/tencent/ksanallm/blob/main/csrc/engine/inference_engine.cpp) (Inference engine), [`python/ksana_llm/ksana_engine.py`](https://github.com/tencent/ksanallm/blob/main/python/ksana_llm/ksana_engine.py) (Python engine), [`csrc/service/inference_service.cpp`](https://github.com/tencent/ksanallm/blob/main/csrc/service/inference_service.cpp) (Inference service), [`csrc/cache_manager/prefix_cache_manager.cpp`](https://github.com/tencent/ksanallm/blob/main/csrc/cache_manager/prefix_cache_manager.cpp) (Prefix cache)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving"]
  node_pyengine["Python engine<br/>[ksana_engine.py]"]
  node_plugins["Plugin processing<br/>[ksana_engine.py]"]
  node_service["Inference service"]
  node_endpoint["Serving endpoint"]
  node_stream["Streaming output<br/>[ksana_engine.py]"]
end

subgraph group_runtime["Inference runtime"]
  node_engine["Inference engine"]
  node_scheduler["Batch scheduler"]
  node_decode["Decoding modes"]
end

subgraph group_memory["KV memory"]
  node_cachemgr["Prefix cache"]
  node_allocator["Block allocator"]
  node_kvblocks[("KV blocks")]
end

subgraph group_distributed["Distributed transfer"]
  node_transfer["Transfer engine"]
  node_pdv2["PD v2 connector"]
  node_mooncake["Mooncake backend"]
end

subgraph group_compute["Compute backends"]
  node_kernels["Optimized kernels"]
  node_ascend["Ascend operations"]
end

node_client(("Client"))
node_hardware{{"GPU / NPU"}}

node_client -->|"submits request"| node_pyengine
node_pyengine -->|"preprocesses input"| node_plugins
node_pyengine -->|"invokes serving"| node_service
node_service -->|"delegates handling"| node_endpoint
node_endpoint -->|"queues inference"| node_engine
node_engine -->|"initializes scheduler"| node_scheduler
node_scheduler -->|"schedules decoding"| node_decode
node_pyengine -->|"returns iterator"| node_stream
node_engine -->|"initializes cache"| node_cachemgr
node_cachemgr -->|"allocates blocks"| node_allocator
node_allocator -->|"manages"| node_kvblocks
node_engine -->|"initializes transfer"| node_transfer
node_engine -->|"creates connector"| node_pdv2
node_pdv2 -.->|"transfers KV data"| node_mooncake
node_scheduler -->|"dispatches compute"| node_kernels
node_kernels -->|"executes on"| node_hardware
node_scheduler -.->|"dispatches operations"| node_ascend
node_ascend -.->|"executes on"| node_hardware

click node_pyengine "https://github.com/tencent/ksanallm/blob/main/python/ksana_llm/ksana_engine.py"
click node_plugins "https://github.com/tencent/ksanallm/blob/main/python/ksana_llm/ksana_engine.py"
click node_service "https://github.com/tencent/ksanallm/blob/main/csrc/service/inference_service.cpp"
click node_endpoint "https://github.com/tencent/ksanallm/blob/main/csrc/service/inference_service.cpp"
click node_stream "https://github.com/tencent/ksanallm/blob/main/python/ksana_llm/ksana_engine.py"
click node_engine "https://github.com/tencent/ksanallm/blob/main/csrc/engine/inference_engine.cpp"
click node_scheduler "https://github.com/tencent/ksanallm/blob/main/csrc/engine/inference_engine.cpp"
click node_decode "https://github.com/tencent/ksanallm/tree/main/csrc/engine"
click node_cachemgr "https://github.com/tencent/ksanallm/blob/main/csrc/cache_manager/prefix_cache_manager.cpp"
click node_allocator "https://github.com/tencent/ksanallm/blob/main/csrc/cache_manager/block_allocator/engine_block_allocator.cpp"
click node_kvblocks "https://github.com/tencent/ksanallm/tree/main/csrc/cache_manager"
click node_transfer "https://github.com/tencent/ksanallm/blob/main/csrc/transfer/transfer_engine.cpp"
click node_pdv2 "https://github.com/tencent/ksanallm/blob/main/csrc/pd_v2/mooncake/mooncake_engine_connector.h"
click node_mooncake "https://github.com/tencent/ksanallm/tree/main/csrc/pd_v2/mooncake"
click node_kernels "https://github.com/tencent/ksanallm/tree/main/3rdparty/LLM_kernels/csrc/kernels/nvidia"
click node_ascend "https://github.com/tencent/ksanallm/tree/main/3rdparty/LLM_kernels/csrc/kernels/ascend"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pyengine,node_plugins,node_service,node_endpoint,node_stream,node_client toneBlue
class node_engine,node_scheduler,node_decode toneAmber
class node_cachemgr,node_allocator,node_kvblocks toneMint
class node_transfer,node_pdv2,node_mooncake toneRose
class node_kernels,node_ascend,node_hardware toneIndigo
```

## Components

### Serving

- [Python engine](https://github.com/tencent/ksanallm/blob/main/python/ksana_llm/ksana_engine.py): `python/ksana_llm/ksana_engine.py`
- [Plugin processing](https://github.com/tencent/ksanallm/blob/main/python/ksana_llm/ksana_engine.py): `python/ksana_llm/ksana_engine.py`
- [Inference service](https://github.com/tencent/ksanallm/blob/main/csrc/service/inference_service.cpp): `csrc/service/inference_service.cpp`
- [Serving endpoint](https://github.com/tencent/ksanallm/blob/main/csrc/service/inference_service.cpp): `csrc/service/inference_service.cpp`
- [Streaming output](https://github.com/tencent/ksanallm/blob/main/python/ksana_llm/ksana_engine.py): `python/ksana_llm/ksana_engine.py`

### Inference runtime

- [Inference engine](https://github.com/tencent/ksanallm/blob/main/csrc/engine/inference_engine.cpp): `csrc/engine/inference_engine.cpp`
- [Batch scheduler](https://github.com/tencent/ksanallm/blob/main/csrc/engine/inference_engine.cpp): `csrc/engine/inference_engine.cpp`
- [Decoding modes](https://github.com/tencent/ksanallm/tree/main/csrc/engine): `csrc/engine`

### KV memory

- [Prefix cache](https://github.com/tencent/ksanallm/blob/main/csrc/cache_manager/prefix_cache_manager.cpp): `csrc/cache_manager/prefix_cache_manager.cpp`
- [Block allocator](https://github.com/tencent/ksanallm/blob/main/csrc/cache_manager/block_allocator/engine_block_allocator.cpp): `csrc/cache_manager/block_allocator/engine_block_allocator.cpp`
- [KV blocks](https://github.com/tencent/ksanallm/tree/main/csrc/cache_manager): `csrc/cache_manager`

### Distributed transfer

- [Transfer engine](https://github.com/tencent/ksanallm/blob/main/csrc/transfer/transfer_engine.cpp): `csrc/transfer/transfer_engine.cpp`
- [PD v2 connector](https://github.com/tencent/ksanallm/blob/main/csrc/pd_v2/mooncake/mooncake_engine_connector.h): `csrc/pd_v2/mooncake/mooncake_engine_connector.h`
- [Mooncake backend](https://github.com/tencent/ksanallm/tree/main/csrc/pd_v2/mooncake): `csrc/pd_v2/mooncake`

### Compute backends

- [Optimized kernels](https://github.com/tencent/ksanallm/tree/main/3rdparty/LLM_kernels/csrc/kernels/nvidia): `3rdparty/LLM_kernels/csrc/kernels/nvidia`
- [Ascend operations](https://github.com/tencent/ksanallm/tree/main/3rdparty/LLM_kernels/csrc/kernels/ascend): `3rdparty/LLM_kernels/csrc/kernels/ascend`

### Other components

- **Client**
- **GPU / NPU**

## Connections

- Client → Python engine: submits request
- Python engine → Plugin processing: preprocesses input
- Python engine → Inference service: invokes serving
- Inference service → Serving endpoint: delegates handling
- Serving endpoint → Inference engine: queues inference
- Inference engine → Batch scheduler: initializes scheduler
- Batch scheduler → Decoding modes: schedules decoding
- Python engine → Streaming output: returns iterator
- Inference engine → Prefix cache: initializes cache
- Prefix cache → Block allocator: allocates blocks
- Block allocator → KV blocks: manages
- Inference engine → Transfer engine: initializes transfer
- Inference engine → PD v2 connector: creates connector
- PD v2 connector → Mooncake backend: transfers KV data
- Batch scheduler → Optimized kernels: dispatches compute
- Optimized kernels → GPU / NPU: executes on
- Batch scheduler → Ascend operations: dispatches operations
- Ascend operations → GPU / NPU: executes on