How modelengine-group/unified-cache-management works
Unified Cache Management (UCM) is a plugin-style library for LLM inference runtimes, not a standalone server. Its architecture is centered on three extensibility boundaries:
1. integration hooks into external model-serving frameworks,
2. algorithm modules for sparse KV reuse/retrieval,
3. storage backends for persisting and reloading KV cache blocks.
The repository is split so that Python coordinates control flow, while performance-critical pieces live in C++/CUDA/Ascend extensions.
Main architectural layers
• Runtime integration layer: ucm/integration
- This is where UCM connects to external inference stacks.
- It contains adapters and patches for:
- vLLM, including separate patch trees for v0.9.2 and v0.11.0
- SGLang
- MindIE LLM
- These modules hook into scheduler, worker, attention, KV connector, and cache-manager code in the upstream runtime.
- The purpose is to inject UCM behavior without replacing the host engine.
• Sparse KV algorithm layer: ucm/sparse
- This is the core algorithmic layer for non-prefix cache reuse.
- It includes several algorithm families:
- blend for cache blend / blockwise RoPE-style handling
- esa for retrieval-based sparse attention
- gsa for generalized sparse attention, with offload and prefetch submodules
- gsa_on_device for device-side sparse retrieval and cache operations
- kvstar for multistep retrieval workflows
- rerope for window extrapolation / RoPE-related attention modifications
- The README’s “UcmSparseBase” concept is reflected here: algorithms are designed to plug into the same hook points in the model runtime.
• Storage and cache persistence layer: ucm/store
- This layer abstracts KV cache persistence and retrieval.
- It is not tied to one storage medium; the repo includes multiple backends:
- posix for local file and AIO-based storage
- nfsstore for POSIX/NFS-mounted shared storage
- ds3fs for a file-system-backed distributed store
- pcstore for prefix-cache-oriented storage
- pipeline for pipeline-oriented store integration
- fake and empty for testing/no-op behavior
- mooncakestore as another connector
- Factories such as ucm/store/factory.py and factory_v1.py select the backend.
- The important boundary here is that sparse algorithms do not talk directly to filesystems or devices; they target a KV store interface.
• Shared native infrastructure: ucm/shared
- This is the cross-cutting foundation used by both sparse and store code.
- It contains:
- logging
- metrics
- thread primitives
- timing and status utilities
- device/data transport abstractions under trans
- The transport layer has device-specific implementations for CUDA, Ascend, Simu, and Maca, which matters because the same cache operations need to move data across different accelerators and host memory.
• PD disaggregation support: ucm/pd
- This is a smaller but important subsystem for storage-compute separation.
- The repo includes a toy_proxy_server.py and supporting docs/examples, indicating that UCM can split prefill/decode responsibilities across machines or roles.
Data flow through the system
1. A user launches a supported inference stack or example.
- Examples in the repo show offline inference, benchmarking, and deployment scripts.
- Supported runtimes include vLLM, SGLang, and MindIE LLM.
2. UCM is injected into the runtime via patch/adaptor modules.
- The integration layer modifies scheduler and attention paths.
- For vLLM, there are patched implementations for model runner, worker, KV connector, scheduler, and attention layers.
- For Ascend/NPU paths, there are separate patch trees and device-specific workers/runners.
3. During prefill or decode, UCM decides whether to:
- reuse prefix KV,
- retrieve cached KV blocks from external storage,
- dump KV blocks out of device memory,
- or apply a sparse attention strategy instead of full dense attention.
- This decision is driven by the active sparse algorithm or KV connector.
4. The selected sparse module computes block selection or retrieval strategy.
- Some algorithms are pure Python orchestration.
- Others delegate to native code for speed, especially when performing top-k selection, hashing, hamming-distance retrieval, or cache reshaping.
5. Cache data is moved through a backend store.
- The storage boundary handles persistence to local files, shared filesystems, or distributed cache layers.
- The same abstraction serves both prefix caching and sparse cache reuse.
6. Device-specific kernels and transport code handle high-throughput movement and compute.
- CUDA and Ascend implementations exist where performance matters.
- This is especially visible in gsa_on_device and the native code under ucm/shared/trans and ucm/store.
Key subsystems and their responsibilities
• ucm/integration/vllm
- The largest integration surface.
- Contains patch files and runtime-adapted modules for vLLM internals.
- Separate branches of code exist for dense prefix caching, sparse attention, and Ascend adaptations.
- This is the main place where UCM becomes “active” inside the model server.
• ucm/integration/sglang
- A lighter integration path with patching and connector logic for SGLang.
• ucm/integration/mindie
- MindIE-specific integration with patching, unified cache memory pool, and a native hash extension.
• ucm/store/*
- Concrete persistence and transfer backends.
- pcstore, nfsstore, and posix are especially central because they represent different deployment topologies: local single-node, shared filesystem, or distributed/cache-separated environments.
• ucm/sparse/gsa_on_device
- Contains the most hardware-specific sparse retrieval path.
- Includes Python orchestration, config files for different models, and large native subtrees for CUDA and Ascend.
- This subsystem is where sparse retrieval can be pushed onto the accelerator instead of being handled purely on CPU or host-side Python.
• ucm/sparse/kvstar
- A retrieval pipeline with explicit task management and a native core.
- The architecture here is task-oriented: query/request preparation, task queueing, worker execution, result waiting, and SIMD compute kernels.
• ucm/shared/metrics and ucm/logger
- These are supporting observability layers.
- They provide the stats needed to understand cache hit/miss behavior, offload performance, and retrieval overhead.
• benchmarks, examples, and test
- These are not production runtime paths, but they matter architecturally because they show how the system is validated.
- Benchmarks cover logging and trace replay.
- Example scripts demonstrate offline inference, deployments, and metrics integration.
- Tests include end-to-end scenarios for offline/online inference, environment precheck, evaluator flows, sparse variants, and connector save/load behavior.
Technology boundaries that matter
• Python is the orchestration and extension layer.
- It owns configuration, integration logic, algorithm selection, and runtime hooks.
• C++ is used for native task engines, storage engines, retrieval engines, and low-level infrastructure.
- Built with CMake, with many subprojects and pybind-style bindings.
• CUDA and Ascend/NPU code exist side by side.
- This is important architecturally because the same sparse-cache ideas are implemented on multiple accelerator families.
• The repo assumes external LLM runtimes as hosts.
- UCM augments vLLM/SGLang/MindIE rather than replacing them.
- That means the diagram should show UCM as a middleware/plugin layer between the application and the model runtime internals.
How to draw the architecture
A clear diagram would show:
• At the top: applications, benchmarks, or inference scripts.
• In the middle: supported runtimes such as vLLM, SGLang, and MindIE.
• Overlaid into those runtimes: UCM integration hooks for scheduler, worker, attention, and KV connector paths.
• Beneath that: UCM’s sparse algorithm modules, with separate branches for blend, ESA, GSA, KVStar, and RoPE-related behavior.
• Side-by-side with algorithms: storage backends for cache persistence and retrieval.
• Under everything: shared C++/CUDA/Ascend transport, logging, and metrics infrastructure.
• Off to one side: PD disaggregation support and on-device sparse retrieval as specialized deployments.
In short, UCM is a modular cache-management and sparse-attention framework that sits inside existing LLM inference engines, routes KV cache through pluggable storage backends, and uses native device-aware components to make prefix caching, sparse attention, and disaggregated inference practical across GPU and Ascend environments.
Main entry points: factory.py Store Factory · base.py Sparse Core · trans Transport · ucm_connector.py UCM Connectors
Components 25
Integration
Adapters and patch trees that inject UCM into external runtimes.
- vLLM · Primary runtime integration surface with patch trees for multiple versions and Ascend variants.
ucm/integration/vllm - SGLang · Integration and connector layer for SGLang.
ucm/integration/sglang - MindIE · Integration and unified-cache support for MindIE LLM.
ucm/integration/mindie - UCM Connectors · Common connector entrypoints that bridge host runtime cache hooks to UCM.
ucm/integration/vllm/ucm_connector.py
Sparse KV
Algorithm modules for cache reuse, sparse attention, and retrieval.
- Sparse Core · Shared sparse-algorithm interface and state used by multiple cache strategies.
ucm/sparse/base.py - Blend · Cache blend and blockwise RoPE-style handling.
ucm/sparse/blend - ESA · Retrieval-based sparse attention module.
ucm/sparse/esa - GSA · Generalized sparse attention with offload and prefetch subsystems.
ucm/sparse/gsa - GSA On-Device · Hardware-specific retrieval and cache reshape path for CUDA and Ascend.
ucm/sparse/gsa_on_device - KVStar · Multistep retrieval workflow with task queueing, waiting, and SIMD compute.
ucm/sparse/kvstar - ReRoPE · RoPE/window extrapolation utilities for modified attention paths.
ucm/sparse/rerope
Storage
Pluggable KV cache persistence and reload backends.
- Store Factory · Selects concrete KV store implementations and versioned APIs.
ucm/store/factory.py - Posix Store · Local file and AIO-backed persistence backend.
ucm/store/posix - NFS Store · Shared filesystem backend with device-aware transfer paths.
ucm/store/nfsstore - PC Store · Prefix-cache-oriented backend for shared-buffer style cache movement.
ucm/store/pcstore - DS3FS · Distributed filesystem-style store implementation.
ucm/store/ds3fs - Pipeline Store · Pipeline-oriented store connector and loader integration.
ucm/store/pipeline - Fake/Empty · No-op and testing stores used for validation and fallback behavior.
ucm/store/fake
Shared Infra
Cross-cutting native logging, metrics, and transport foundations.
- Transport · Shared device/host buffer transport and stream abstractions across CUDA, Ascend, Simu, and Maca.
ucm/shared/trans - Metrics · Metrics collection and export used to track cache and retrieval performance.
ucm/shared/metrics - Logging · Shared logger foundation for native and Python-facing components.
ucm/shared/infra/logger
Usage
Examples, benchmarks, and test flows that exercise the library.
- Examples · Offline inference, deployment, and metrics examples showing how UCM is launched.
examples - Benchmarks · Replay and logging performance entrypoints used to evaluate cache behavior.
benchmarks - Tests · End-to-end and component coverage for sparse, store, and connector flows.
test
PD Support
Prefill/decode disaggregation support.
- PD Disagg · Prefill/decode separation support, including proxy server and deployment patterns.
ucm/pd
Connections 37
- Examples → vLLM: launches · Example entrypoints demonstrate vLLM deployment and offline inference.
- Examples → SGLang: launches · Example entrypoints demonstrate SGLang deployment.
- Examples → MindIE: launches · Example entrypoints demonstrate MindIE deployment.
- Benchmarks → Metrics: measures · Benchmarks inspect logging and trace performance.
- Tests → Store Factory: validates · End-to-end tests cover store selection and save/load behavior.
- Tests → GSA On-Device: validates · Sparse tests exercise device-side retrieval and cache reshape paths.
- vLLM → UCM Connectors: hooks · vLLM patch trees connect host runtime internals to UCM.
- SGLang → UCM Connectors: hooks · SGLang integration uses connector glue for cache control.
- MindIE → UCM Connectors: hooks · MindIE integration routes through UCM connector logic.
- UCM Connectors → Sparse Core: dispatches · Runtime hooks hand control to the active sparse algorithm.
- Sparse Core → Blend: selects · Sparse core can instantiate blend-style cache reuse.
- Sparse Core → ESA: selects · Sparse core can instantiate retrieval-based sparse attention.
- Sparse Core → GSA: selects · Sparse core can instantiate generalized sparse attention.
- Sparse Core → GSA On-Device: selects · Sparse core can route to device-side retrieval kernels.
- Sparse Core → KVStar: selects · Sparse core can instantiate multistep retrieval workflows.
- Sparse Core → ReRoPE: selects · Sparse core can apply RoPE/window extrapolation modifications.
- GSA → Store Factory: loads from · GSA offload/prefetch paths use the KV store abstraction.
- ESA → Store Factory: retrieves from · Retrieval-based sparse attention pulls cached KV through the store layer.
- KVStar → Store Factory: retrieves from · Retrieval tasks load blocks via store backends.
- Store Factory → Posix Store: chooses · Factory can instantiate local POSIX/AIO storage.
- Store Factory → NFS Store: chooses · Factory can instantiate shared filesystem storage.
- Store Factory → PC Store: chooses · Factory can instantiate prefix-cache storage.
- Store Factory → DS3FS: chooses · Factory can instantiate distributed filesystem storage.
- Store Factory → Pipeline Store: chooses · Factory can instantiate pipeline-oriented storage connectors.
- Store Factory → Fake/Empty: chooses · Factory can instantiate no-op or fake stores for fallback and testing.
- Posix Store → Transport: moves data via · POSIX storage relies on shared transport and buffer primitives.
- NFS Store → Transport: moves data via · NFS backend uses device-aware transport and buffers.
- PC Store → Transport: moves data via · PC store uses shared transport abstractions.
- DS3FS → Transport: moves data via · Distributed store shares transport infrastructure.
- Pipeline Store → Transport: moves data via · Pipeline store loads native transport components.
- GSA On-Device → Transport: executes on · Device-side sparse retrieval uses accelerator transport and kernels.
- vLLM → Metrics: emits · Runtime patches expose cache and scheduler metrics.
- Store Factory → Metrics: emits · Store layers contribute cache persistence metrics.
- vLLM → Logging: uses · Integration patches rely on shared logging.
- Store Factory → Logging: uses · Store implementations use shared logging infrastructure.
- PD Disagg → Store Factory: separates with · PD disaggregation relies on cache persistence and reload boundaries.
- PD Disagg → UCM Connectors: coordinates · Proxy/disaggregated flows coordinate through the UCM integration layer.