How paddlepaddle/fastdeploy works
1. What this repository is
FastDeploy is a production inference/deployment stack for large language models and vision-language models, centered on PaddlePaddle but also supporting Torch-format models. The codebase is organized as a full serving system, not just a model runtime: it includes request ingress, scheduling, worker execution, custom kernels, cache transport, observability, benchmarks, and hardware-specific adaptations.
2. Top-level request lifecycle
A request typically enters through a CLI or HTTP API entrypoint, is normalized into an internal request object, routed to a scheduler, assigned to one or more workers, executed by a model runner, and then streamed back through output processors. For online serving, the same flow can be exposed as OpenAI-compatible endpoints or through a router layer that sits in front of model instances.
3. Serving and API surface
The main entrypoint layer lives under fastdeploy/entrypoints and fastdeploy/router. This includes:
- CLI commands for chat, completion, serve, batch run, tokenizer, and environment collection
- OpenAI-compatible APIs and V1 variants
- a router launch path
- request/response shaping, usage accounting, and tool-parser hooks
This layer is the boundary between external clients and the internal engine.
4. Engine, scheduler, and worker split
The core runtime is split across fastdeploy/engine, fastdeploy/scheduler, and fastdeploy/worker.
- The engine owns request objects, sampling params, async orchestration, and resource management.
- The scheduler handles batching, placement, data-parallel or splitwise scheduling, and resource allocation policies.
- The worker layer performs actual model execution on a specific device/backend.
This separation is important for understanding scale-out: control-plane decisions are decoupled from device execution.
5. Model execution stack
fastdeploy/model_executor contains the model-facing implementation: model loaders, model definitions, layers, logits processors, sampling, quantization, and graph optimization. It is where model-specific behavior is adapted into a common runtime interface. This layer also contains support for:
- dense text models
- multimodal models
- MoE models
- MTP/speculative decoding models
- reasoning and tool-calling parsers
6. Input preprocessing and multimodal handling
Preprocessing lives under fastdeploy/input and fastdeploy/multimodal. These modules convert raw user input into model-ready tensors and metadata:
- text tokenization and chat formatting
- image/video/audio preprocessing
- VL model-specific processors for ERNIE, Qwen, PaddleOCR, etc.
This forms the front half of the data pipeline before a request reaches the model runner.
7. Output processing and token streaming
The fastdeploy/output package handles postprocessing, token stream assembly, and transport of generated tokens back to the caller. It works with the engine/worker boundary to support streaming responses, top-k/logprob payloads, and final response packaging. This is also where generation-time metadata is transformed into API-friendly output.
8. Distributed communication and inter-process coordination
fastdeploy/inter_communicator and fastdeploy/distributed provide the messaging backbone between processes and components. The repository uses a mix of IPC, ZMQ, FMQ, and custom all-reduce paths to connect engine workers, cache managers, and router/service processes. This is a key architectural boundary: inference is not purely in-process; it is split across cooperating services and shared-memory/message-passing channels.
9. KV cache management and external cache transfer
fastdeploy/cache_manager is dedicated to cache lifecycle and movement. It includes prefix cache managers, multimodal cache handling, metrics, and transfer factories for IPC, file-store, RDMA, and storage-backed cache paths. This subsystem is central to performance because FastDeploy emphasizes KV cache reuse, offload, and transfer between disaggregated components.
10. Hardware/platform abstraction
fastdeploy/platforms and the backend-specific code under fastdeploy/model_executor/layers/backends adapt the stack to multiple accelerators:
- CUDA/NVIDIA
- XPU
- GCU
- DCU
- Iluvatar
- Intel HPU
- Metax/Maca/NPU variants
The architecture is intentionally multi-backend: core model logic is shared, while attention, MoE, quantization, and kernel choices vary by platform.
11. Custom ops and kernel acceleration
custom_ops is a major subsystem and contains the performance-critical device kernels. It includes GPU/XPU/Iluvatar/Metax-specific operators for:
- attention variants
- KV-cache write/read and cache movement
- MoE dispatch/reduce/expert FFN
- quantized GEMM and dequantization
- sampling, penalties, and generation control
- speculative decoding and MTP-specific operations
These kernels are what make the higher-level engine performant across different quantization and deployment modes.
12. Router and multi-service deployment
The repository has both Python and Go router implementations. The Go router under fastdeploy/golang_router appears to handle instance registration, health, metrics, load-balancing strategies, and gateway routing. This is the outer control plane for multi-instance deployments, supporting load-aware dispatch and router-level scheduling decisions. The Python router path under fastdeploy/router provides the launch and integration side.
13. Advanced inference features as first-class subsystems
Several features are built in rather than bolted on:
- speculative decoding and MTP under fastdeploy/spec_decode
- prefix caching and global cache pooling
- chunked prefill
- weight update / online quantization
- reasoning budgets and structured outputs
- disaggregated PD deployment
- RL/rollout support under fastdeploy/rl
These features affect the architecture because they change how work is scheduled, how cache is reused, and how intermediate model roles are split.
14. Observability and operational tooling
Logging, metrics, and tracing are first-class:
- fastdeploy/logger
- fastdeploy/metrics
- fastdeploy/trace
The examples and docs show integration with Prometheus and OpenTelemetry. This indicates the system is designed for production serving with metrics-driven routing and performance analysis.
15. Benchmarks, examples, and validation
The repo includes extensive benchmarks, examples, and tests. Benchmarks cover serving throughput/latency, model-specific YAML configs, and hardware-specific scenarios. Tests are broad and mirror the architecture: engine, scheduler, input processing, model executor, quantization, operators, router, cache, distributed communication, and end-to-end serving. The breadth of tests reflects that correctness depends on cross-layer interactions, not just isolated modules.
16. How to draw the architecture
A good diagram would show:
- clients entering through CLI/OpenAI/API/router
- request normalization and preprocessing
- engine + scheduler control plane
- worker processes per device/backend
- model executor calling backend-specific ops/custom kernels
- KV cache and inter-process communication as a parallel plane
- output streaming back to the client
- metrics/logging/tracing surrounding the whole stack
The key boundary in this repo is between the orchestration/control plane and the device-execution/performance plane, with cache movement and custom kernels bridging them.
Main entry points: model_executor Model Executor · entrypoints Entrypoints · engine Engine · tests Tests
Components 20
Client Surface
CLI, HTTP APIs, and router launch paths exposed to users and external services.
- Entrypoints · User-facing commands, OpenAI-compatible servers, batch entrypoints, and request shaping.
fastdeploy/entrypoints - Router · Python-side router launch and integration layer for serving deployments.
fastdeploy/router - Go Router · Multi-instance gateway, registration, health, metrics, and load-balancing control plane.
fastdeploy/golang_router
Control Plane
Request orchestration, scheduling, worker coordination, and distributed messaging.
- Engine · Owns request objects, async control flow, resource management, and engine-side request state.
fastdeploy/engine - Scheduler · Handles batching, placement, scheduling policy, and worker/resource coordination.
fastdeploy/scheduler - Workers · Device-specific worker processes and model runners that execute inference on accelerators.
fastdeploy/worker - IPC & ZMQ · Inter-process queues, signals, ZMQ links, and FMQ transport between engine, router, and workers.
fastdeploy/inter_communicator
Runtime & Models
Model execution, preprocessing, postprocessing, platform adapters, and plugins.
- Model Executor · Model loaders, layer implementations, logits/sampling logic, quantization, and graph optimization.
fastdeploy/model_executor - Input Pipeline · Text, chat, multimodal, and tokenizer preprocessing into model-ready inputs.
fastdeploy/input - Multimodal · Shared multimodal primitives and registries used by VL and media-heavy models.
fastdeploy/multimodal - Output Pipeline · Token assembly, response packaging, and output transfer back to callers.
fastdeploy/output - Platforms · Hardware abstraction for CPU, CUDA, XPU, GCU, DCU, Iluvatar, HPU, Maca, and NPU targets.
fastdeploy/platforms - Plugins · Hook points for custom input processors, model registration, model runners, token processing, and parsing.
fastdeploy/plugins
Performance Plane
KV cache movement, custom kernels, and backend-specific acceleration.
- Cache Manager · Prefix cache, multimodal cache, cache metrics, and transfer factories for IPC, RDMA, and storage-backed paths.
fastdeploy/cache_manager - Custom Ops · Performance-critical operators for attention, MoE, sampling, cache movement, quantization, and speculative decoding.
custom_ops - Distributed · Custom all-reduce and distributed communication utilities used by multi-process inference.
fastdeploy/distributed - Spec Decode · Speculative decoding strategies including MTP, n-gram, and suffix-based proposal/verification.
fastdeploy/spec_decode
Ops & Validation
Observability, benchmarks, examples, and tests that exercise the serving stack.
- Observability · Logging, metrics, tracing, and middleware surrounding serving requests.
fastdeploy/metrics - Benchmarks · Serving throughput, latency, tokenizer, FMQ, and model-specific benchmarking scripts and configs.
benchmarks - Tests · Broad unit, integration, end-to-end, hardware-specific, and determinism coverage across the stack.
tests
Connections 28
- Entrypoints → Engine: creates requests · CLI and API entrypoints normalize incoming calls into engine request objects.
- Entrypoints → Input Pipeline: preprocesses · User-facing surfaces feed text and multimodal inputs into preprocessing.
- Entrypoints → Output Pipeline: formats responses · OpenAI and CLI surfaces rely on output packaging and streaming response helpers.
- Entrypoints → Router: launches · Serving entrypoints can start or integrate with the Python router path.
- Router → Go Router: pairs with · Python launch integration complements the Go router deployment path.
- Go Router → Scheduler: dispatches · Gateway-side load balancing and instance management feed scheduler decisions indirectly.
- Engine → Scheduler: submits work · Engine orchestration hands requests to the scheduler for batching and placement.
- Scheduler → Workers: assigns · Scheduling decisions place batches onto worker processes.
- Workers → Model Executor: runs model · Worker processes invoke model execution stacks on the selected device/backend.
- Model Executor → Input Pipeline: consumes · Model execution expects preprocessed tensors and metadata from the input pipeline.
- Model Executor → Output Pipeline: emits tokens · Generated tokens and metadata are turned into streaming outputs.
- Model Executor → Platforms: adapts to · Execution paths select backend-specific implementations through platform abstraction.
- Model Executor → Plugins: extends via · Model behavior can be customized through plugin registration and parser hooks.
- Model Executor → Spec Decode: supports · Speculative decoding integrates with model execution and sampling.
- Model Executor → Custom Ops: calls kernels · Layers and backend implementations rely on custom ops for hot paths.
- Cache Manager → Workers: serves cache · Workers read and write KV cache state through cache managers and transfer paths.
- Cache Manager → IPC & ZMQ: moves over · Cache transfer uses IPC and message-passing channels for disaggregated components.
- IPC & ZMQ → Engine: coordinates · IPC links connect engine-side control flow with workers and cache services.
- Distributed → Workers: connects · Distributed collectives underpin multi-process worker execution.
- Custom Ops → Cache Manager: accelerates · Custom kernels implement cache write/read, swap, and transfer hot paths.
- Custom Ops → Distributed: implements · Custom all-reduce and communication kernels live in the distributed performance plane.
- Observability → Entrypoints: wraps · Logging, metrics, and tracing surround external request handling.
- Observability → Engine: instruments · Engine, scheduler, and worker flows expose metrics and traces.
- Benchmarks → Entrypoints: measures · Benchmark harnesses exercise CLI, API, and serving paths.
- Tests → Engine: validates · Tests cover request orchestration and resource management.
- Tests → Scheduler: validates · Tests cover batching, placement, and scheduling policy behavior.
- Tests → Model Executor: validates · Tests cover layers, loaders, quantization, and model-specific execution.
- Tests → Cache Manager: validates · Tests cover prefix cache, RDMA transfer, and cache manager behavior.