How tencent/ksanallm works
KsanaLLM is an LLM inference and serving engine. A request enters through the Python engine, which prepares input and runs plugin preprocessing before invoking serving; the C++ service delegates handling to an endpoint and the inference engine manages runtime execution. The documented product also includes streaming and multiple decoding methods, prefix caching, multi-GPU execution, and GPU/NPU kernels. Source excerpts show service, engine, cache, and transfer responsibilities; several capability-specific connections are based on the README and tree, since their implementation was not sampled.
Main entry points: inference_engine.cpp Inference engine · ksana_engine.py Python engine · inference_service.cpp Inference service · prefix_cache_manager.cpp Prefix cache
Components 18
Serving
- Python engine
python/ksana_llm/ksana_engine.py - Plugin processing
python/ksana_llm/ksana_engine.py - Inference service
csrc/service/inference_service.cpp - Serving endpoint
csrc/service/inference_service.cpp - Streaming output
python/ksana_llm/ksana_engine.py
Inference runtime
- Inference engine
csrc/engine/inference_engine.cpp - Batch scheduler
csrc/engine/inference_engine.cpp - Decoding modes
csrc/engine
KV memory
- Prefix cache
csrc/cache_manager/prefix_cache_manager.cpp - Block allocator
csrc/cache_manager/block_allocator/engine_block_allocator.cpp - KV blocks
csrc/cache_manager
Distributed transfer
- Transfer engine
csrc/transfer/transfer_engine.cpp - PD v2 connector
csrc/pd_v2/mooncake/mooncake_engine_connector.h - Mooncake backend
csrc/pd_v2/mooncake
Compute backends
- Optimized kernels
3rdparty/LLM_kernels/csrc/kernels/nvidia - Ascend operations
3rdparty/LLM_kernels/csrc/kernels/ascend
Other components
- Client
- GPU / NPU
Connections 18
- Client → Python engine: submits request
- Python engine → Plugin processing: preprocesses input
- Python engine → Inference service: invokes serving
- Inference service → Serving endpoint: delegates handling
- Serving endpoint → Inference engine: queues inference
- Inference engine → Batch scheduler: initializes scheduler
- Batch scheduler → Decoding modes: schedules decoding
- Python engine → Streaming output: returns iterator
- Inference engine → Prefix cache: initializes cache
- Prefix cache → Block allocator: allocates blocks
- Block allocator → KV blocks: manages
- Inference engine → Transfer engine: initializes transfer
- Inference engine → PD v2 connector: creates connector
- PD v2 connector → Mooncake backend: transfers KV data
- Batch scheduler → Optimized kernels: dispatches compute
- Optimized kernels → GPU / NPU: executes on
- Batch scheduler → Ascend operations: dispatches operations
- Ascend operations → GPU / NPU: executes on