How wtdcode/vllm-backport works
vLLM Backport is an LLM inference server fork aimed at running frontier models and quantization formats on older GPUs. The primary workflow is a client request entering the HTTP API, being processed by the engine, and producing generated output; the README also highlights model-specific decoding, caching, multimodal support, and distributed execution. The graph preserves those product branches. Source excerpts are partial, so some subsystem links reflect documented capabilities rather than inspected call sites.
Main entry points: engine Engine core · routes HTTP API · llm_engine.py LLM engine · engine.py Engine inputs
Components 18
Serving
- HTTP API
rust/src/server/src/routes - Engine inputs
vllm/inputs/engine.py - LLM engine
vllm/v1/engine/llm_engine.py - Metrics routes
rust/src/server/src/routes/metrics.rs
Inference
- Engine core
vllm/v1/engine - Streaming parsers
vllm/parser/engine/streaming_parser_engine.py - Model tool parsers
vllm/tool_parsers - Tokenization
rust/src/tokenizer/src/lib.rs - Multimodal inputs
vllm/config/multimodal.py - KV cache config
vllm/config/cache.py - Speculative decoding
vllm/config/speculative.py
Acceleration
- Quantized kernels
csrc/libtorch_stable/quantization - Graph compilation
vllm/compilation - Distributed execution
vllm/distributed - Weight transfer
vllm/distributed/weight_transfer/sharded_rdt_engine.py
Extensions
- LMCache service
Other components
- Client
- Generated response
Connections 17
- Client → HTTP API: sends request
- HTTP API → Engine inputs: normalizes input
- HTTP API → LLM engine: submits request
- LLM engine → Engine core: dispatches work
- Engine core → Generated response: returns output
- LLM engine → Streaming parsers: parses streamed output
- Streaming parsers → Model tool parsers: selects format parser
- Engine inputs → Tokenization: uses token IDs
- Engine inputs → Multimodal inputs: carries media data
- Engine core → KV cache config: configures KV cache
- Engine core → Speculative decoding: configures decoding
- Engine core → Quantized kernels: uses kernels
- Engine core → Graph compilation: compiles execution
- Engine core → Distributed execution: executes across workers
- LLM engine → Weight transfer: updates weights
- Engine core → LMCache service: transfers KV blocks
- HTTP API → Metrics routes: exposes metrics