How ericlbuehler/candle-vllm works
1. What this repo is
Candle-vLLM is a Rust inference/serving system for local LLMs. It ships both a library crate (src/lib.rs) and a command-line server binary (src/main.rs) that can expose an OpenAI-compatible API, stream tokens, and optionally serve a built-in UI.
2. Top-level runtime flow
The main request path is: client request → OpenAI API server (src/openai/openai_server.rs) → request normalization/types (src/openai/requests.rs, responses.rs, sampling_params.rs) → pipeline/engine orchestration (src/openai/pipelines/*) → scheduler (src/scheduler/*) → backend model execution (src/backend/*) → streamed or buffered response back to the client.
3. API and compatibility boundary
The openai module is the public protocol boundary. It translates OpenAI-style chat/completion/streaming requests into internal model inputs, applies conversation state handling (src/openai/conversation/*), and returns OpenAI-shaped responses. This is the main integration surface for external tools, agents, and UI clients.
4. Model serving core
The serving logic is split into reusable pipeline variants under src/openai/pipelines/: threaded, multi-process, streaming, and multi-process/distributed helpers. These components coordinate batching, token generation, and request lifecycle management through the llm_engine and pipeline abstractions.
5. Scheduler and KV/cache management
The scheduler subsystem (src/scheduler/) is responsible for continuous batching, sequence tracking, and memory-aware execution. block_engine.rs, cache_engine.rs, sequence.rs, and prefix_cache.rs indicate that the system manages KV blocks explicitly, supports prefix reuse, and adapts to constrained memory budgets. mamba.rs suggests model-specific scheduling support where needed.
6. Backend execution layer
The backend module is where model weights are loaded and executed. It contains loaders/adapters for GGUF, GPTQ/Marlin, and general graph execution (gguf.rs, gptq.rs, graph.rs), plus runtime services such as cache, heartbeat, and progress reporting. backend/custom_ops/* contains specialized kernels/operators, including MoE and sorting paths.
7. Model family abstraction
src/openai/models/* is a large model-specific layer with implementations for many architectures: Llama, Qwen, Mistral, Gemma, DeepSeek, GLM, Phi, Yi, StableLM, MiniMax, and several vision-language variants. These modules encapsulate architecture-specific layer wiring, rotary embeddings, attention variants, MLPs, MoE, and quantized weight formats.
8. Quantization and device-specific optimization
The repo supports multiple execution formats and kernels: BF16/F16, GGUF, GPTQ/Marlin, FP8 blockwise models, MXFP4/NVFP4, and “TurboQuant” KV compression. The architecture reflects this through specialized model files (quantized_*), layer helpers (quantized_var_builder.rs, linear.rs, rotary_emb.rs), and backend features for CUDA, Metal, flash attention, FlashInfer, and CUDA graphs.
9. Multimodal path
Multimodal support is handled through dedicated model subtrees like gemma3_vl, llama4, mistral3_vl, and qwen3_vl, plus src/openai/multimodal.rs. These modules add vision input/config/vision-tower logic while still flowing through the same serving pipeline and response surface as text-only models.
10. Tool calling and MCP
Tool integration is a separate subsystem under src/tools/* and src/mcp/*. The tools package parses and validates model-emitted tool calls, while the MCP modules implement client/server/transport/manager pieces for Model Context Protocol integration. This sits between generation output parsing and downstream tool execution.
11. Streaming and output post-processing
Token streaming is explicitly supported via src/openai/streaming.rs, pipelines/streaming.rs, and communicator.rs. After model decoding, output can be incrementally parsed/logged, including tool-call extraction (src/tools/stream_parser.rs) and model-specific logits/postprocessing (logits_processor.rs, logger.rs).
12. Distributed and multi-node execution
The repo includes explicit distributed inference support in src/openai/distributed.rs and src/backend/heartbeat.rs, with CLI flags for multi-GPU and TCP-based multi-node coordination. The architecture assumes local weight availability per node and uses NCCL/bootstrap-style coordination rather than MPI.
13. Packaging and deployment surfaces
The repo is packaged for Rust/Cargo, Docker, and Python-based conversion/utilities. build.rs, Dockerfile, build_docker.sh, .github/workflows/*, setup.py, and the examples/ directory support release builds, container deployment, format conversion, benchmarking, and client-side usage examples.
14. How to draw the architecture
A useful diagram would show three boundaries: external clients/UI/MCP tools at the top, the OpenAI-compatible server and pipelines in the middle, and the scheduler/backend/model kernels at the bottom. Side paths should show optional multimodal input, tool-calling/MCP, prefix/KV cache services, and distributed execution across multiple devices or nodes.
Main entry points: openai_server.rs OpenAI server · pipelines/mod.rs Pipelines · scheduler/mod.rs Scheduler · backend/mod.rs Backend core
Components 31
Clients & entrypoints
External-facing entrypoints and optional UI/deployment surfaces.
- Server binary · CLI entrypoint that starts serving and runtime coordination.
src/main.rs - Library crate · Reusable crate surface exported to binaries and downstream users.
src/lib.rs
OpenAI boundary
Protocol translation, request handling, pipelines, and serving orchestration.
- API surface · High-level API glue and public-facing server wiring.
src/api.rs - OpenAI server · OpenAI-compatible request handling and response delivery.
src/openai/openai_server.rs - Request types · Chat/completion request normalization and sampling inputs.
src/openai/requests.rs - Response types · OpenAI-shaped response structures and streaming payloads.
src/openai/responses.rs - Pipelines · Serving pipeline variants and engine coordination layer.
src/openai/pipelines/mod.rs - LLM engine · Shared execution engine used by pipeline variants.
src/openai/pipelines/llm_engine.rs - Threaded pipe · Single-process threaded serving path.
src/openai/pipelines/threaded.rs - Multi-process pipe · Process-separated serving path for scaling and isolation.
src/openai/pipelines/multiprocess.rs - Streaming pipe · Incremental token delivery and stream orchestration.
src/openai/pipelines/streaming.rs - Conversation state · Conversation assembly and default prompt/state handling.
src/openai/conversation/mod.rs - Distributed mode · Multi-node / multi-GPU serving coordination hooks.
src/openai/distributed.rs - Multimodal glue · Bridges image and other non-text inputs into the OpenAI pipeline.
src/openai/multimodal.rs - Streaming output · Token streaming and incremental response formatting.
src/openai/streaming.rs
Scheduling & cache
Sequence tracking, batching, and KV/cache-aware execution.
- Scheduler · Sequence scheduling, continuous batching, and memory-aware execution.
src/scheduler/mod.rs - Block engine · Explicit KV block management and allocation strategy.
src/scheduler/block_engine.rs - Cache engine · Cache lifecycle and reuse management for inference.
src/scheduler/cache_engine.rs - Sequence tracking · Per-request token/sequence bookkeeping and scheduling state.
src/scheduler/sequence.rs - Prefix cache · Prefix reuse and prompt cache acceleration path.
src/scheduler/prefix_cache.rs
Model backend
Weight loading, graph execution, runtime services, and custom kernels.
- Backend core · Model loading, execution, graph runtime, and backend services.
src/backend/mod.rs - GGUF loader · GGUF model format loading and backend adaptation.
src/backend/gguf.rs - GPTQ/Marlin · GPTQ and Marlin quantized execution support.
src/backend/gptq.rs - Graph runtime · Core graph execution and model runtime wiring.
src/backend/graph.rs - Custom ops · Specialized backend kernels and operators.
src/backend/custom_ops/mod.rs
Model families
Architecture-specific model implementations and shared layer helpers.
- Shared layers · Reusable attention, MLP, norm, rotary, and quantized layer helpers.
src/openai/models/layers/mod.rs - Llama family · Representative decoder-only model implementation and variants.
src/openai/models/llama.rs - Qwen VL · Vision-language model subtree for Qwen-style multimodal inference.
src/openai/models/qwen3_vl/mod.rs - Gemma VL · Vision-language model subtree for Gemma-style multimodal inference.
src/openai/models/gemma3_vl/mod.rs
Tools & MCP
Tool parsing/execution integration and MCP client/server plumbing.
- Tool parsing · Tool-call parsing, validation, and schema helpers.
src/tools/mod.rs - MCP stack · Model Context Protocol client/server/transport/manager plumbing.
src/mcp/mod.rs
Connections 32
- Server binary → OpenAI server: starts · Binary launches the OpenAI-compatible server.
- Library crate → API surface: exports · Library crate exposes the API/server modules.
- API surface → OpenAI server: wires · Public API connects to serving entrypoints.
- OpenAI server → Request types: parses · Server normalizes incoming OpenAI-style requests.
- OpenAI server → Response types: returns · Server emits OpenAI-shaped responses and streams.
- OpenAI server → Pipelines: dispatches · Request handling delegates into serving pipelines.
- OpenAI server → Conversation state: state · Conversation context is assembled before generation.
- OpenAI server → Multimodal glue: optional input · Vision and other non-text inputs are adapted here.
- OpenAI server → Streaming output: streams · Streaming responses are surfaced to clients.
- Pipelines → LLM engine: hosts · Pipelines share the common engine abstraction.
- Pipelines → Threaded pipe: variant · Single-process serving path.
- Pipelines → Multi-process pipe: variant · Multi-process serving path.
- Pipelines → Streaming pipe: variant · Incremental decoding path.
- Pipelines → Distributed mode: optional · Distributed coordination is enabled when configured.
- LLM engine → Scheduler: drives · Engine feeds work into the scheduler.
- Scheduler → Block engine: allocates · Scheduler manages KV block placement.
- Scheduler → Cache engine: reuses · Scheduler consults cache-aware execution state.
- Scheduler → Sequence tracking: tracks · Per-request sequence state is maintained.
- Scheduler → Prefix cache: optimizes · Prefix reuse accelerates repeated prompts.
- Scheduler → Backend core: executes via · Scheduled work is handed to backend runtime.
- Backend core → GGUF loader: loads · GGUF weights are adapted into runtime form.
- Backend core → GPTQ/Marlin: loads · Quantized model formats are supported.
- Backend core → Graph runtime: runs · Graph execution performs model forward passes.
- Backend core → Custom ops: uses · Specialized kernels accelerate model execution.
- Graph runtime → Shared layers: composes · Graph execution relies on shared layer building blocks.
- Shared layers → Llama family: used by · Representative model family built on shared layers.
- Shared layers → Qwen VL: used by · Multimodal Qwen subtree builds on shared layers.
- Shared layers → Gemma VL: used by · Multimodal Gemma subtree builds on shared layers.
- Qwen VL → Multimodal glue: ingests · Vision-language inputs are normalized into model inputs.
- Gemma VL → Multimodal glue: ingests · Vision-language inputs are normalized into model inputs.
- OpenAI server → Tool parsing: tool calls · Model-emitted tool calls are parsed and validated.
- Tool parsing → MCP stack: bridges · Tool integration can hand off to MCP clients/servers.