# paddlepaddle/fastdeploy architecture

> Architecture diagram of the GitHub repository paddlepaddle/fastdeploy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/paddlepaddle/fastdeploy
- Repository: https://github.com/paddlepaddle/fastdeploy

## Overview

1. **What this repository is**
   FastDeploy is a production inference/deployment stack for large language models and vision-language models, centered on PaddlePaddle but also supporting Torch-format models. The codebase is organized as a full serving system, not just a model runtime: it includes request ingress, scheduling, worker execution, custom kernels, cache transport, observability, benchmarks, and hardware-specific adaptations.

2. **Top-level request lifecycle**
   A request typically enters through a CLI or HTTP API entrypoint, is normalized into an internal request object, routed to a scheduler, assigned to one or more workers, executed by a model runner, and then streamed back through output processors. For online serving, the same flow can be exposed as OpenAI-compatible endpoints or through a router layer that sits in front of model instances.

3. **Serving and API surface**
   The main entrypoint layer lives under `fastdeploy/entrypoints` and `fastdeploy/router`. This includes:
   - CLI commands for chat, completion, serve, batch run, tokenizer, and environment collection
   - OpenAI-compatible APIs and V1 variants
   - a router launch path
   - request/response shaping, usage accounting, and tool-parser hooks  
   This layer is the boundary between external clients and the internal engine.

4. **Engine, scheduler, and worker split**
   The core runtime is split across `fastdeploy/engine`, `fastdeploy/scheduler`, and `fastdeploy/worker`.
   - The **engine** owns request objects, sampling params, async orchestration, and resource management.
   - The **scheduler** handles batching, placement, data-parallel or splitwise scheduling, and resource allocation policies.
   - The **worker** layer performs actual model execution on a specific device/backend.  
   This separation is important for understanding scale-out: control-plane decisions are decoupled from device execution.

5. **Model execution stack**
   `fastdeploy/model_executor` contains the model-facing implementation: model loaders, model definitions, layers, logits processors, sampling, quantization, and graph optimization. It is where model-specific behavior is adapted into a common runtime interface. This layer also contains support for:
   - dense text models
   - multimodal models
   - MoE models
   - MTP/speculative decoding models
   - reasoning and tool-calling parsers

6. **Input preprocessing and multimodal handling**
   Preprocessing lives under `fastdeploy/input` and `fastdeploy/multimodal`. These modules convert raw user input into model-ready tensors and metadata:
   - text tokenization and chat formatting
   - image/video/audio preprocessing
   - VL model-specific processors for ERNIE, Qwen, PaddleOCR, etc.  
   This forms the front half of the data pipeline before a request reaches the model runner.

7. **Output processing and token streaming**
   The `fastdeploy/output` package handles postprocessing, token stream assembly, and transport of generated tokens back to the caller. It works with the engine/worker boundary to support streaming responses, top-k/logprob payloads, and final response packaging. This is also where generation-time metadata is transformed into API-friendly output.

8. **Distributed communication and inter-process coordination**
   `fastdeploy/inter_communicator` and `fastdeploy/distributed` provide the messaging backbone between processes and components. The repository uses a mix of IPC, ZMQ, FMQ, and custom all-reduce paths to connect engine workers, cache managers, and router/service processes. This is a key architectural boundary: inference is not purely in-process; it is split across cooperating services and shared-memory/message-passing channels.

9. **KV cache management and external cache transfer**
   `fastdeploy/cache_manager` is dedicated to cache lifecycle and movement. It includes prefix cache managers, multimodal cache handling, metrics, and transfer factories for IPC, file-store, RDMA, and storage-backed cache paths. This subsystem is central to performance because FastDeploy emphasizes KV cache reuse, offload, and transfer between disaggregated components.

10. **Hardware/platform abstraction**
   `fastdeploy/platforms` and the backend-specific code under `fastdeploy/model_executor/layers/backends` adapt the stack to multiple accelerators:
   - CUDA/NVIDIA
   - XPU
   - GCU
   - DCU
   - Iluvatar
   - Intel HPU
   - Metax/Maca/NPU variants  
   The architecture is intentionally multi-backend: core model logic is shared, while attention, MoE, quantization, and kernel choices vary by platform.

11. **Custom ops and kernel acceleration**
   `custom_ops` is a major subsystem and contains the performance-critical device kernels. It includes GPU/XPU/Iluvatar/Metax-specific operators for:
   - attention variants
   - KV-cache write/read and cache movement
   - MoE dispatch/reduce/expert FFN
   - quantized GEMM and dequantization
   - sampling, penalties, and generation control
   - speculative decoding and MTP-specific operations  
   These kernels are what make the higher-level engine performant across different quantization and deployment modes.

12. **Router and multi-service deployment**
   The repository has both Python and Go router implementations. The Go router under `fastdeploy/golang_router` appears to handle instance registration, health, metrics, load-balancing strategies, and gateway routing. This is the outer control plane for multi-instance deployments, supporting load-aware dispatch and router-level scheduling decisions. The Python router path under `fastdeploy/router` provides the launch and integration side.

13. **Advanced inference features as first-class subsystems**
   Several features are built in rather than bolted on:
   - speculative decoding and MTP under `fastdeploy/spec_decode`
   - prefix caching and global cache pooling
   - chunked prefill
   - weight update / online quantization
   - reasoning budgets and structured outputs
   - disaggregated PD deployment
   - RL/rollout support under `fastdeploy/rl`  
   These features affect the architecture because they change how work is scheduled, how cache is reused, and how intermediate model roles are split.

14. **Observability and operational tooling**
   Logging, metrics, and tracing are first-class:
   - `fastdeploy/logger`
   - `fastdeploy/metrics`
   - `fastdeploy/trace`  
   The examples and docs show integration with Prometheus and OpenTelemetry. This indicates the system is designed for production serving with metrics-driven routing and performance analysis.

15. **Benchmarks, examples, and validation**
   The repo includes extensive `benchmarks`, `examples`, and `tests`. Benchmarks cover serving throughput/latency, model-specific YAML configs, and hardware-specific scenarios. Tests are broad and mirror the architecture: engine, scheduler, input processing, model executor, quantization, operators, router, cache, distributed communication, and end-to-end serving. The breadth of tests reflects that correctness depends on cross-layer interactions, not just isolated modules.

16. **How to draw the architecture**
   A good diagram would show:
   - clients entering through CLI/OpenAI/API/router
   - request normalization and preprocessing
   - engine + scheduler control plane
   - worker processes per device/backend
   - model executor calling backend-specific ops/custom kernels
   - KV cache and inter-process communication as a parallel plane
   - output streaming back to the client
   - metrics/logging/tracing surrounding the whole stack  
   The key boundary in this repo is between the orchestration/control plane and the device-execution/performance plane, with cache movement and custom kernels bridging them.

Main entry points: [`fastdeploy/model_executor`](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/model_executor) (Model Executor), [`fastdeploy/entrypoints`](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/entrypoints) (Entrypoints), [`fastdeploy/engine`](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/engine) (Engine), [`tests`](https://github.com/paddlepaddle/fastdeploy/tree/develop/tests) (Tests)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Client Surface"]
  node_entrypoints["Entrypoints<br/>CLI + HTTP APIs"]
  node_router["Router<br/>Python launcher"]
  node_golang_router["Go Router<br/>gateway + LB"]
end

subgraph group_control["Control Plane"]
  node_engine["Engine<br/>request orchestration"]
  node_scheduler["Scheduler<br/>batching + placement"]
  node_workers["Workers<br/>device executors"]
  node_ipc["IPC & ZMQ<br/>messaging backbone"]
end

subgraph group_runtime["Runtime & Models"]
  node_model_executor["Model Executor<br/>model runtime"]
  node_input["Input Pipeline<br/>preprocess"]
  node_multimodal["Multimodal<br/>image/video/audio"]
  node_output["Output Pipeline<br/>streaming response"]
  node_platforms["Platforms<br/>backend abstraction"]
  node_plugins["Plugins<br/>extension points"]
end

subgraph group_perf["Performance Plane"]
  node_cache[("Cache Manager<br/>KV cache lifecycle")]
  node_ops["Custom Ops<br/>device kernels"]
  node_dist["Distributed<br/>collectives + comms"]
  node_spec_decode["Spec Decode<br/>draft verification"]
end

subgraph group_ops["Ops & Validation"]
  node_observability["Observability"]
  node_benchmarks["Benchmarks<br/>performance harness"]
  node_tests["Tests<br/>validation suite"]
end

node_entrypoints -->|"creates requests"| node_engine
node_entrypoints -->|"preprocesses"| node_input
node_entrypoints -->|"formats responses"| node_output
node_entrypoints -->|"launches"| node_router
node_router -->|"pairs with"| node_golang_router
node_golang_router -->|"dispatches"| node_scheduler
node_engine -->|"submits work"| node_scheduler
node_scheduler -->|"assigns"| node_workers
node_workers -->|"runs model"| node_model_executor
node_model_executor -->|"consumes"| node_input
node_model_executor -->|"emits tokens"| node_output
node_model_executor -->|"adapts to"| node_platforms
node_model_executor -->|"extends via"| node_plugins
node_model_executor -->|"supports"| node_spec_decode
node_model_executor -->|"calls kernels"| node_ops
node_cache -->|"serves cache"| node_workers
node_cache -->|"moves over"| node_ipc
node_ipc -->|"coordinates"| node_engine
node_dist -->|"connects"| node_workers
node_ops -->|"accelerates"| node_cache
node_ops -->|"implements"| node_dist
node_observability -->|"wraps"| node_entrypoints
node_observability -->|"instruments"| node_engine
node_benchmarks -->|"measures"| node_entrypoints
node_tests -->|"validates"| node_engine
node_tests -->|"validates"| node_scheduler
node_tests -->|"validates"| node_model_executor
node_tests -->|"validates"| node_cache

click node_entrypoints "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/entrypoints"
click node_router "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/router"
click node_golang_router "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/golang_router"
click node_engine "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/engine"
click node_scheduler "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/scheduler"
click node_workers "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/worker"
click node_ipc "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/inter_communicator"
click node_model_executor "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/model_executor"
click node_input "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/input"
click node_multimodal "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/multimodal"
click node_output "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/output"
click node_platforms "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/platforms"
click node_plugins "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/plugins"
click node_cache "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/cache_manager"
click node_ops "https://github.com/paddlepaddle/fastdeploy/tree/develop/custom_ops"
click node_dist "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/distributed"
click node_spec_decode "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/spec_decode"
click node_observability "https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/metrics"
click node_benchmarks "https://github.com/paddlepaddle/fastdeploy/tree/develop/benchmarks"
click node_tests "https://github.com/paddlepaddle/fastdeploy/tree/develop/tests"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_entrypoints,node_router,node_golang_router toneBlue
class node_engine,node_scheduler,node_workers,node_ipc toneAmber
class node_model_executor,node_input,node_multimodal,node_output,node_platforms,node_plugins toneMint
class node_cache,node_ops,node_dist,node_spec_decode toneRose
class node_observability,node_benchmarks,node_tests toneIndigo
```

## Components

### Client Surface

CLI, HTTP APIs, and router launch paths exposed to users and external services.

- [Entrypoints](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/entrypoints): CLI + HTTP APIs · User-facing commands, OpenAI-compatible servers, batch entrypoints, and request shaping. · `fastdeploy/entrypoints`
- [Router](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/router): Python launcher · Python-side router launch and integration layer for serving deployments. · `fastdeploy/router`
- [Go Router](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/golang_router): gateway + LB · Multi-instance gateway, registration, health, metrics, and load-balancing control plane. · `fastdeploy/golang_router`

### Control Plane

Request orchestration, scheduling, worker coordination, and distributed messaging.

- [Engine](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/engine): request orchestration · Owns request objects, async control flow, resource management, and engine-side request state. · `fastdeploy/engine`
- [Scheduler](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/scheduler): batching + placement · Handles batching, placement, scheduling policy, and worker/resource coordination. · `fastdeploy/scheduler`
- [Workers](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/worker): device executors · Device-specific worker processes and model runners that execute inference on accelerators. · `fastdeploy/worker`
- [IPC & ZMQ](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/inter_communicator): messaging backbone · Inter-process queues, signals, ZMQ links, and FMQ transport between engine, router, and workers. · `fastdeploy/inter_communicator`

### Runtime & Models

Model execution, preprocessing, postprocessing, platform adapters, and plugins.

- [Model Executor](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/model_executor): model runtime · Model loaders, layer implementations, logits/sampling logic, quantization, and graph optimization. · `fastdeploy/model_executor`
- [Input Pipeline](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/input): preprocess · Text, chat, multimodal, and tokenizer preprocessing into model-ready inputs. · `fastdeploy/input`
- [Multimodal](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/multimodal): image/video/audio · Shared multimodal primitives and registries used by VL and media-heavy models. · `fastdeploy/multimodal`
- [Output Pipeline](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/output): streaming response · Token assembly, response packaging, and output transfer back to callers. · `fastdeploy/output`
- [Platforms](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/platforms): backend abstraction · Hardware abstraction for CPU, CUDA, XPU, GCU, DCU, Iluvatar, HPU, Maca, and NPU targets. · `fastdeploy/platforms`
- [Plugins](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/plugins): extension points · Hook points for custom input processors, model registration, model runners, token processing, and parsing. · `fastdeploy/plugins`

### Performance Plane

KV cache movement, custom kernels, and backend-specific acceleration.

- [Cache Manager](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/cache_manager): KV cache lifecycle · Prefix cache, multimodal cache, cache metrics, and transfer factories for IPC, RDMA, and storage-backed paths. · `fastdeploy/cache_manager`
- [Custom Ops](https://github.com/paddlepaddle/fastdeploy/tree/develop/custom_ops): device kernels · Performance-critical operators for attention, MoE, sampling, cache movement, quantization, and speculative decoding. · `custom_ops`
- [Distributed](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/distributed): collectives + comms · Custom all-reduce and distributed communication utilities used by multi-process inference. · `fastdeploy/distributed`
- [Spec Decode](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/spec_decode): draft verification · Speculative decoding strategies including MTP, n-gram, and suffix-based proposal/verification. · `fastdeploy/spec_decode`

### Ops & Validation

Observability, benchmarks, examples, and tests that exercise the serving stack.

- [Observability](https://github.com/paddlepaddle/fastdeploy/tree/develop/fastdeploy/metrics): logs + metrics + trace · Logging, metrics, tracing, and middleware surrounding serving requests. · `fastdeploy/metrics`
- [Benchmarks](https://github.com/paddlepaddle/fastdeploy/tree/develop/benchmarks): performance harness · Serving throughput, latency, tokenizer, FMQ, and model-specific benchmarking scripts and configs. · `benchmarks`
- [Tests](https://github.com/paddlepaddle/fastdeploy/tree/develop/tests): validation suite · Broad unit, integration, end-to-end, hardware-specific, and determinism coverage across the stack. · `tests`

## Connections

- Entrypoints → Engine: creates requests · CLI and API entrypoints normalize incoming calls into engine request objects.
- Entrypoints → Input Pipeline: preprocesses · User-facing surfaces feed text and multimodal inputs into preprocessing.
- Entrypoints → Output Pipeline: formats responses · OpenAI and CLI surfaces rely on output packaging and streaming response helpers.
- Entrypoints → Router: launches · Serving entrypoints can start or integrate with the Python router path.
- Router → Go Router: pairs with · Python launch integration complements the Go router deployment path.
- Go Router → Scheduler: dispatches · Gateway-side load balancing and instance management feed scheduler decisions indirectly.
- Engine → Scheduler: submits work · Engine orchestration hands requests to the scheduler for batching and placement.
- Scheduler → Workers: assigns · Scheduling decisions place batches onto worker processes.
- Workers → Model Executor: runs model · Worker processes invoke model execution stacks on the selected device/backend.
- Model Executor → Input Pipeline: consumes · Model execution expects preprocessed tensors and metadata from the input pipeline.
- Model Executor → Output Pipeline: emits tokens · Generated tokens and metadata are turned into streaming outputs.
- Model Executor → Platforms: adapts to · Execution paths select backend-specific implementations through platform abstraction.
- Model Executor → Plugins: extends via · Model behavior can be customized through plugin registration and parser hooks.
- Model Executor → Spec Decode: supports · Speculative decoding integrates with model execution and sampling.
- Model Executor → Custom Ops: calls kernels · Layers and backend implementations rely on custom ops for hot paths.
- Cache Manager → Workers: serves cache · Workers read and write KV cache state through cache managers and transfer paths.
- Cache Manager → IPC & ZMQ: moves over · Cache transfer uses IPC and message-passing channels for disaggregated components.
- IPC & ZMQ → Engine: coordinates · IPC links connect engine-side control flow with workers and cache services.
- Distributed → Workers: connects · Distributed collectives underpin multi-process worker execution.
- Custom Ops → Cache Manager: accelerates · Custom kernels implement cache write/read, swap, and transfer hot paths.
- Custom Ops → Distributed: implements · Custom all-reduce and communication kernels live in the distributed performance plane.
- Observability → Entrypoints: wraps · Logging, metrics, and tracing surround external request handling.
- Observability → Engine: instruments · Engine, scheduler, and worker flows expose metrics and traces.
- Benchmarks → Entrypoints: measures · Benchmark harnesses exercise CLI, API, and serving paths.
- Tests → Engine: validates · Tests cover request orchestration and resource management.
- Tests → Scheduler: validates · Tests cover batching, placement, and scheduling policy behavior.
- Tests → Model Executor: validates · Tests cover layers, loaders, quantization, and model-specific execution.
- Tests → Cache Manager: validates · Tests cover prefix cache, RDMA transfer, and cache manager behavior.
