How jd-opensource/xllm works
1. What this repo is
xLLM is a distributed inference framework for large models, with a strong focus on Chinese AI accelerators. Its architecture is explicitly service-engine decoupled: request handling and scheduling are separated from execution on the inference engine.
2. Top-level flow
The main request path is: client/API entrypoint → service layer parsing/normalization → scheduler → runtime worker/executor → model layers → hardware kernels → streamed or batched response back to the caller. The architecture is built to support both online and offline serving.
3. Service layer boundary
The xllm/api_service, xllm/server, xllm/pybind, xllm/c_api, and xllm/cc_api directories are the outward-facing interfaces. They expose different protocols and bindings for chat/completion/embedding/rerank/image-generation style workloads, while sharing the same underlying runtime. This is the main boundary between external users and the engine.
4. Scheduling subsystem
xllm/core/scheduler contains the policy layer that decides how requests are admitted and grouped. It includes chunked prefill, continuous decoding, fixed-step, zero-eviction, disaggregated PD, and DIT-specific schedulers. This layer is central to throughput/latency tradeoffs and is where batching, prioritization, and request splitting happen.
5. Runtime and worker execution
xllm/core/runtime is the execution orchestration layer. It contains worker implementations for LLM, VLM, embedding, speculative decoding, MTP, suffix, and DIT flows, plus executor abstractions and client-side RPC orchestration. This layer bridges scheduler decisions to actual model execution.
6. Distributed runtime
xllm/core/distributed_runtime handles multi-process and multi-node execution. It includes master/worker roles, remote worker management, communication channels, and specialized services for disaggregated PD, PD out-of-core, speculative execution, LLM, REC, and VLM variants. This is the distributed control plane for scaling inference beyond one process or one device.
7. Model framework
xllm/core/framework provides the shared model runtime data structures: requests, sequences, sampling state, batch construction, tokenization, chat templates, model context, loaders, and quant/model arguments. It also owns memory-oriented subsystems such as KV cache, block managers, prefix cache, and xtensor-backed virtual/physical page management.
8. Model families
xllm/models defines supported model architectures and their registry. The codebase covers LLM, VLM, REC, and DIT/image-generation families, with many model-specific adapters for Qwen, DeepSeek, GLM, Llama, Oxygen, and related variants. These are the model definitions that sit above the generic framework but below device-specific execution.
9. Layer implementations
xllm/core/layers contains reusable neural network building blocks and model-specific layers. There are common implementations for attention, MLP, MoE, rotary/positional embedding, norms, and vision components, plus device-specialized versions under cuda, npu, mlu, musa, and ilu. This is where high-level model graphs are translated into executable layer logic.
10. Kernel and operator layer
xllm/core/kernels is the low-level performance boundary. It hosts device kernels for attention, matmul, norm, MoE, sampling, cache reshaping, quantization, graph capture, and fused ops. The repo has separate implementations per hardware backend, reflecting that portability is achieved by reimplementing the hot path per accelerator family.
11. Memory and cache management
KV cache is a first-class subsystem, not an implementation detail. The framework includes hierarchical KV cache transfer, prefix cache, embedding cache, global cache, and Mooncake-based offload/prefetch support. Memory management also spans xtensor page allocators and shared virtual memory abstractions, which are important for diagramming data movement and storage boundaries.
12. Hardware abstraction
xllm/core/platform and xllm/core/kernels together define the device abstraction layer. The project supports CUDA, NPU, MLU, MUSA, and ILU, with backend-specific process groups, graph executors, streams, and synchronization helpers. The architecture is therefore multi-backend rather than GPU-only.
13. Compiler/tooling pipeline
xllm/compiler/tilelang is a build-time kernel generation pipeline, especially for Ascend-related flows and custom kernel families. It produces device kernels and integrates with toolchains and patches, so compilation is part of the architecture, not just a developer convenience. The repo also includes scripts for packaging, benchmarking, and hardware-specific builds.
14. External dependencies that shape the design
The repo relies on brpc for high-performance service RPC, etcd for coordination/discovery, safetensors for model weight loading, and tokenizer libraries for fast tokenization. These external systems matter because they define the service boundary, model-loading path, and distributed orchestration model.
15. Testing and operational support
Tests are spread across schedulers, kernels, layers, loaders, and protocol services, indicating that correctness is validated at multiple layers of the stack. Documentation and deployment scripts show that the intended deployment model is production inference serving rather than a library-only project.
Main entry points: kernels Kernels · runtime Runtime · framework Framework · api_service API service
Components 19
Service boundary
External APIs and server entrypoints for serving requests.
- API service · Chat, completion, embedding, rerank, and image-generation service implementations.
xllm/api_service - Server · Top-level server process and registry for serving endpoints.
xllm/server - Python binding · Python-facing bindings and convenience wrappers for model/service use.
xllm/pybind - C API · C-facing API and examples for embedding xLLM in native applications.
xllm/c_api - C++ API · C++ API surface and examples for direct integration and service requests.
xllm/cc_api
Orchestration core
Scheduling, runtime, and distributed execution control.
- Scheduler · Chunked prefill, continuous decoding, fixed-step, zero-eviction, disaggregated PD, and DIT scheduling policies.
xllm/core/scheduler - Runtime · Executor and worker implementations that bridge scheduling to model execution.
xllm/core/runtime - Distributed runtime · Master/worker coordination, remote workers, and distributed services for scaling inference.
xllm/core/distributed_runtime
Model stack
Shared model framework, supported families, and request/data structures.
- Framework · Shared request, sequence, tokenizer, batch, sampling, cache, loader, and model-context structures.
xllm/core/framework - Model registry · Supported LLM, VLM, REC, and DIT model definitions and registry.
xllm/models - Layers · Reusable attention, MLP, MoE, normalization, rotary, and vision layer implementations.
xllm/core/layers
Device backends
Device-specific layers and kernels across supported accelerators.
- Platform · Device, stream, synchronization, and memory abstractions shared across backends.
xllm/core/platform - Kernels · Low-level device kernels for attention, matmul, sampling, cache reshaping, quantization, and fused ops.
xllm/core/kernels - CUDA backend · CUDA-specific kernels and layer/runtime implementations.
xllm/core/kernels/cuda - NPU backend · NPU-specific kernels and layer/runtime implementations, including tilelang integration.
xllm/core/kernels/npu - MLU backend · MLU-specific kernels and layer implementations.
xllm/core/kernels/mlu - MUSA backend · MUSA-specific kernels and layer implementations.
xllm/core/kernels/musa - ILU backend · ILU-specific kernels and layer implementations.
xllm/core/kernels/ilu
Build tooling
Compiler and packaging pipeline for generated or specialized kernels.
- TileLang compiler · Build-time kernel generation and target-specific compilation pipeline.
xllm/compiler/tilelang
Connections 19
- API service → Scheduler: admit/normalize · Requests are parsed and converted into schedulable units.
- Server → API service: host APIs · Server process exposes the service layer.
- Python binding → Runtime: invoke · Python bindings reach the underlying runtime and workers.
- C API → Runtime: invoke · C API calls enter the runtime execution path.
- C++ API → Runtime: invoke · C++ API calls enter the runtime execution path.
- Scheduler → Runtime: dispatch · Scheduling decisions drive worker execution.
- Runtime → Distributed runtime: scale out · Runtime uses distributed control when multi-process or multi-node execution is needed.
- Distributed runtime → Framework: serve batches · Distributed workers exchange requests, sequences, and shared state through framework structures.
- Runtime → Framework: build inputs · Workers consume framework requests, batches, sampling state, and caches.
- Framework → Model registry: instantiate · Model loaders and context create concrete model families.
- Model registry → Layers: compose · Model definitions are assembled from reusable layers.
- Layers → Kernels: lower to ops · Layers rely on backend kernels for performance-critical operations.
- Kernels → Platform: use device APIs · Kernel execution depends on platform streams, device, and memory abstractions.
- CUDA backend → Kernels: implements · CUDA provides one kernel backend.
- NPU backend → Kernels: implements · NPU provides one kernel backend.
- MLU backend → Kernels: implements · MLU provides one kernel backend.
- MUSA backend → Kernels: implements · MUSA provides one kernel backend.
- ILU backend → Kernels: implements · ILU provides one kernel backend.
- TileLang compiler → NPU backend: generate · Compiler tooling generates specialized NPU kernels and build artifacts.