# jd-opensource/xllm architecture

> Architecture diagram of the GitHub repository jd-opensource/xllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jd-opensource/xllm
- Repository: https://github.com/jd-opensource/xllm

## Overview

1. **What this repo is**
   xLLM is a distributed inference framework for large models, with a strong focus on Chinese AI accelerators. Its architecture is explicitly **service-engine decoupled**: request handling and scheduling are separated from execution on the inference engine.

2. **Top-level flow**
   The main request path is: client/API entrypoint → service layer parsing/normalization → scheduler → runtime worker/executor → model layers → hardware kernels → streamed or batched response back to the caller. The architecture is built to support both online and offline serving.

3. **Service layer boundary**
   The `xllm/api_service`, `xllm/server`, `xllm/pybind`, `xllm/c_api`, and `xllm/cc_api` directories are the outward-facing interfaces. They expose different protocols and bindings for chat/completion/embedding/rerank/image-generation style workloads, while sharing the same underlying runtime. This is the main boundary between external users and the engine.

4. **Scheduling subsystem**
   `xllm/core/scheduler` contains the policy layer that decides how requests are admitted and grouped. It includes chunked prefill, continuous decoding, fixed-step, zero-eviction, disaggregated PD, and DIT-specific schedulers. This layer is central to throughput/latency tradeoffs and is where batching, prioritization, and request splitting happen.

5. **Runtime and worker execution**
   `xllm/core/runtime` is the execution orchestration layer. It contains worker implementations for LLM, VLM, embedding, speculative decoding, MTP, suffix, and DIT flows, plus executor abstractions and client-side RPC orchestration. This layer bridges scheduler decisions to actual model execution.

6. **Distributed runtime**
   `xllm/core/distributed_runtime` handles multi-process and multi-node execution. It includes master/worker roles, remote worker management, communication channels, and specialized services for disaggregated PD, PD out-of-core, speculative execution, LLM, REC, and VLM variants. This is the distributed control plane for scaling inference beyond one process or one device.

7. **Model framework**
   `xllm/core/framework` provides the shared model runtime data structures: requests, sequences, sampling state, batch construction, tokenization, chat templates, model context, loaders, and quant/model arguments. It also owns memory-oriented subsystems such as KV cache, block managers, prefix cache, and xtensor-backed virtual/physical page management.

8. **Model families**
   `xllm/models` defines supported model architectures and their registry. The codebase covers LLM, VLM, REC, and DIT/image-generation families, with many model-specific adapters for Qwen, DeepSeek, GLM, Llama, Oxygen, and related variants. These are the model definitions that sit above the generic framework but below device-specific execution.

9. **Layer implementations**
   `xllm/core/layers` contains reusable neural network building blocks and model-specific layers. There are common implementations for attention, MLP, MoE, rotary/positional embedding, norms, and vision components, plus device-specialized versions under `cuda`, `npu`, `mlu`, `musa`, and `ilu`. This is where high-level model graphs are translated into executable layer logic.

10. **Kernel and operator layer**
   `xllm/core/kernels` is the low-level performance boundary. It hosts device kernels for attention, matmul, norm, MoE, sampling, cache reshaping, quantization, graph capture, and fused ops. The repo has separate implementations per hardware backend, reflecting that portability is achieved by reimplementing the hot path per accelerator family.

11. **Memory and cache management**
   KV cache is a first-class subsystem, not an implementation detail. The framework includes hierarchical KV cache transfer, prefix cache, embedding cache, global cache, and Mooncake-based offload/prefetch support. Memory management also spans xtensor page allocators and shared virtual memory abstractions, which are important for diagramming data movement and storage boundaries.

12. **Hardware abstraction**
   `xllm/core/platform` and `xllm/core/kernels` together define the device abstraction layer. The project supports CUDA, NPU, MLU, MUSA, and ILU, with backend-specific process groups, graph executors, streams, and synchronization helpers. The architecture is therefore multi-backend rather than GPU-only.

13. **Compiler/tooling pipeline**
   `xllm/compiler/tilelang` is a build-time kernel generation pipeline, especially for Ascend-related flows and custom kernel families. It produces device kernels and integrates with toolchains and patches, so compilation is part of the architecture, not just a developer convenience. The repo also includes scripts for packaging, benchmarking, and hardware-specific builds.

14. **External dependencies that shape the design**
   The repo relies on `brpc` for high-performance service RPC, `etcd` for coordination/discovery, `safetensors` for model weight loading, and tokenizer libraries for fast tokenization. These external systems matter because they define the service boundary, model-loading path, and distributed orchestration model.

15. **Testing and operational support**
   Tests are spread across schedulers, kernels, layers, loaders, and protocol services, indicating that correctness is validated at multiple layers of the stack. Documentation and deployment scripts show that the intended deployment model is production inference serving rather than a library-only project.

Main entry points: [`xllm/core/kernels`](https://github.com/jd-opensource/xllm/tree/main/xllm/core/kernels) (Kernels), [`xllm/core/runtime`](https://github.com/jd-opensource/xllm/tree/main/xllm/core/runtime) (Runtime), [`xllm/core/framework`](https://github.com/jd-opensource/xllm/tree/main/xllm/core/framework) (Framework), [`xllm/api_service`](https://github.com/jd-opensource/xllm/tree/main/xllm/api_service) (API service)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_service["Service boundary"]
  node_api_service["API service<br/>protocol services"]
  node_server["Server<br/>service host"]
  node_pybind["Python binding<br/>pybind API"]
  node_c_api["C API<br/>c bindings"]
  node_cc_api["C++ API<br/>cpp bindings"]
end

subgraph group_orchestration["Orchestration core"]
  node_scheduler["Scheduler<br/>admission policy"]
  node_runtime["Runtime<br/>worker orchestration"]
  node_distributed_runtime["Distributed runtime<br/>multi-node control"]
end

subgraph group_model["Model stack"]
  node_framework["Framework<br/>model runtime"]
  node_models["Model registry<br/>model families"]
  node_layers["Layers<br/>nn building blocks"]
end

subgraph group_backend["Device backends"]
  node_platform["Platform<br/>device abstraction"]
  node_kernels["Kernels<br/>hot path ops"]
  node_cuda["CUDA backend"]
  node_npu["NPU backend"]
  node_mlu["MLU backend"]
  node_musa["MUSA backend"]
  node_ilu["ILU backend"]
end

subgraph group_tooling["Build tooling"]
  node_tilelang["TileLang compiler<br/>kernel generation"]
end

node_api_service -->|"admit/normalize"| node_scheduler
node_server -->|"host APIs"| node_api_service
node_pybind -->|"invoke"| node_runtime
node_c_api -->|"invoke"| node_runtime
node_cc_api -->|"invoke"| node_runtime
node_scheduler -->|"dispatch"| node_runtime
node_runtime -->|"scale out"| node_distributed_runtime
node_distributed_runtime -->|"serve batches"| node_framework
node_runtime -->|"build inputs"| node_framework
node_framework -->|"instantiate"| node_models
node_models -->|"compose"| node_layers
node_layers -->|"lower to ops"| node_kernels
node_kernels -->|"use device APIs"| node_platform
node_cuda -->|"implements"| node_kernels
node_npu -->|"implements"| node_kernels
node_mlu -->|"implements"| node_kernels
node_musa -->|"implements"| node_kernels
node_ilu -->|"implements"| node_kernels
node_tilelang -->|"generate"| node_npu

click node_api_service "https://github.com/jd-opensource/xllm/tree/main/xllm/api_service"
click node_server "https://github.com/jd-opensource/xllm/tree/main/xllm/server"
click node_pybind "https://github.com/jd-opensource/xllm/tree/main/xllm/pybind"
click node_c_api "https://github.com/jd-opensource/xllm/tree/main/xllm/c_api"
click node_cc_api "https://github.com/jd-opensource/xllm/tree/main/xllm/cc_api"
click node_scheduler "https://github.com/jd-opensource/xllm/tree/main/xllm/core/scheduler"
click node_runtime "https://github.com/jd-opensource/xllm/tree/main/xllm/core/runtime"
click node_distributed_runtime "https://github.com/jd-opensource/xllm/tree/main/xllm/core/distributed_runtime"
click node_framework "https://github.com/jd-opensource/xllm/tree/main/xllm/core/framework"
click node_models "https://github.com/jd-opensource/xllm/tree/main/xllm/models"
click node_layers "https://github.com/jd-opensource/xllm/tree/main/xllm/core/layers"
click node_platform "https://github.com/jd-opensource/xllm/tree/main/xllm/core/platform"
click node_kernels "https://github.com/jd-opensource/xllm/tree/main/xllm/core/kernels"
click node_cuda "https://github.com/jd-opensource/xllm/tree/main/xllm/core/kernels/cuda"
click node_npu "https://github.com/jd-opensource/xllm/tree/main/xllm/core/kernels/npu"
click node_mlu "https://github.com/jd-opensource/xllm/tree/main/xllm/core/kernels/mlu"
click node_musa "https://github.com/jd-opensource/xllm/tree/main/xllm/core/kernels/musa"
click node_ilu "https://github.com/jd-opensource/xllm/tree/main/xllm/core/kernels/ilu"
click node_tilelang "https://github.com/jd-opensource/xllm/tree/main/xllm/compiler/tilelang"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api_service,node_server,node_pybind,node_c_api,node_cc_api toneBlue
class node_scheduler,node_runtime,node_distributed_runtime toneAmber
class node_framework,node_models,node_layers toneMint
class node_platform,node_kernels,node_cuda,node_npu,node_mlu,node_musa,node_ilu toneRose
class node_tilelang toneIndigo
```

## Components

### Service boundary

External APIs and server entrypoints for serving requests.

- [API service](https://github.com/jd-opensource/xllm/tree/main/xllm/api_service): protocol services · Chat, completion, embedding, rerank, and image-generation service implementations. · `xllm/api_service`
- [Server](https://github.com/jd-opensource/xllm/tree/main/xllm/server): service host · Top-level server process and registry for serving endpoints. · `xllm/server`
- [Python binding](https://github.com/jd-opensource/xllm/tree/main/xllm/pybind): pybind API · Python-facing bindings and convenience wrappers for model/service use. · `xllm/pybind`
- [C API](https://github.com/jd-opensource/xllm/tree/main/xllm/c_api): c bindings · C-facing API and examples for embedding xLLM in native applications. · `xllm/c_api`
- [C++ API](https://github.com/jd-opensource/xllm/tree/main/xllm/cc_api): cpp bindings · C++ API surface and examples for direct integration and service requests. · `xllm/cc_api`

### Orchestration core

Scheduling, runtime, and distributed execution control.

- [Scheduler](https://github.com/jd-opensource/xllm/tree/main/xllm/core/scheduler): admission policy · Chunked prefill, continuous decoding, fixed-step, zero-eviction, disaggregated PD, and DIT scheduling policies. · `xllm/core/scheduler`
- [Runtime](https://github.com/jd-opensource/xllm/tree/main/xllm/core/runtime): worker orchestration · Executor and worker implementations that bridge scheduling to model execution. · `xllm/core/runtime`
- [Distributed runtime](https://github.com/jd-opensource/xllm/tree/main/xllm/core/distributed_runtime): multi-node control · Master/worker coordination, remote workers, and distributed services for scaling inference. · `xllm/core/distributed_runtime`

### Model stack

Shared model framework, supported families, and request/data structures.

- [Framework](https://github.com/jd-opensource/xllm/tree/main/xllm/core/framework): model runtime · Shared request, sequence, tokenizer, batch, sampling, cache, loader, and model-context structures. · `xllm/core/framework`
- [Model registry](https://github.com/jd-opensource/xllm/tree/main/xllm/models): model families · Supported LLM, VLM, REC, and DIT model definitions and registry. · `xllm/models`
- [Layers](https://github.com/jd-opensource/xllm/tree/main/xllm/core/layers): nn building blocks · Reusable attention, MLP, MoE, normalization, rotary, and vision layer implementations. · `xllm/core/layers`

### Device backends

Device-specific layers and kernels across supported accelerators.

- [Platform](https://github.com/jd-opensource/xllm/tree/main/xllm/core/platform): device abstraction · Device, stream, synchronization, and memory abstractions shared across backends. · `xllm/core/platform`
- [Kernels](https://github.com/jd-opensource/xllm/tree/main/xllm/core/kernels): hot path ops · Low-level device kernels for attention, matmul, sampling, cache reshaping, quantization, and fused ops. · `xllm/core/kernels`
- [CUDA backend](https://github.com/jd-opensource/xllm/tree/main/xllm/core/kernels/cuda): cuda backend · CUDA-specific kernels and layer/runtime implementations. · `xllm/core/kernels/cuda`
- [NPU backend](https://github.com/jd-opensource/xllm/tree/main/xllm/core/kernels/npu): npu backend · NPU-specific kernels and layer/runtime implementations, including tilelang integration. · `xllm/core/kernels/npu`
- [MLU backend](https://github.com/jd-opensource/xllm/tree/main/xllm/core/kernels/mlu): mlu backend · MLU-specific kernels and layer implementations. · `xllm/core/kernels/mlu`
- [MUSA backend](https://github.com/jd-opensource/xllm/tree/main/xllm/core/kernels/musa): musa backend · MUSA-specific kernels and layer implementations. · `xllm/core/kernels/musa`
- [ILU backend](https://github.com/jd-opensource/xllm/tree/main/xllm/core/kernels/ilu): ilu backend · ILU-specific kernels and layer implementations. · `xllm/core/kernels/ilu`

### Build tooling

Compiler and packaging pipeline for generated or specialized kernels.

- [TileLang compiler](https://github.com/jd-opensource/xllm/tree/main/xllm/compiler/tilelang): kernel generation · Build-time kernel generation and target-specific compilation pipeline. · `xllm/compiler/tilelang`

## Connections

- API service → Scheduler: admit/normalize · Requests are parsed and converted into schedulable units.
- Server → API service: host APIs · Server process exposes the service layer.
- Python binding → Runtime: invoke · Python bindings reach the underlying runtime and workers.
- C API → Runtime: invoke · C API calls enter the runtime execution path.
- C++ API → Runtime: invoke · C++ API calls enter the runtime execution path.
- Scheduler → Runtime: dispatch · Scheduling decisions drive worker execution.
- Runtime → Distributed runtime: scale out · Runtime uses distributed control when multi-process or multi-node execution is needed.
- Distributed runtime → Framework: serve batches · Distributed workers exchange requests, sequences, and shared state through framework structures.
- Runtime → Framework: build inputs · Workers consume framework requests, batches, sampling state, and caches.
- Framework → Model registry: instantiate · Model loaders and context create concrete model families.
- Model registry → Layers: compose · Model definitions are assembled from reusable layers.
- Layers → Kernels: lower to ops · Layers rely on backend kernels for performance-critical operations.
- Kernels → Platform: use device APIs · Kernel execution depends on platform streams, device, and memory abstractions.
- CUDA backend → Kernels: implements · CUDA provides one kernel backend.
- NPU backend → Kernels: implements · NPU provides one kernel backend.
- MLU backend → Kernels: implements · MLU provides one kernel backend.
- MUSA backend → Kernels: implements · MUSA provides one kernel backend.
- ILU backend → Kernels: implements · ILU provides one kernel backend.
- TileLang compiler → NPU backend: generate · Compiler tooling generates specialized NPU kernels and build artifacts.
