How w2315516642/vllm works
vLLM is an LLM inference and serving system. The principal workflow accepts a generation request through a serving API, converts it to engine input, runs inference, and streams results back; chat rendering and output parsing support chat, reasoning, and tool calls. The graph also preserves documented model execution, multimodal, distributed, and optimization capabilities. Some runtime wiring is inferred from documented capabilities and the tree; sampled excerpts are limited, so uncertain call edges are omitted.
Main entry points: v1 Engine core · client.rs Engine client · mod.rs Chat rendering · generate.rs Inference server
Components 19
Serving and APIs
- Inference server
rust/src/server/src/routes/inference/generate.rs - OpenAI routes
rust/src/server/src/routes/openai/chat_completions.rs - Chat rendering
rust/src/chat/src/renderer/mod.rs - Stream parsing
vllm/parser/engine/streaming_parser_engine.py
Request and Engine
- Engine inputs
vllm/inputs/engine.py - LLM engine
vllm/v1/engine/llm_engine.py - Engine client
rust/src/engine-core-client/src/client.rs - Engine core
vllm/v1
Model Execution
- Tokenization
rust/src/tokenizer/src/lib.rs - Text processing
rust/src/text/src/lib.rs - Multimodal inputs
- KV cache
vllm/config/cache.py - Inference kernels
csrc/ops.h
Optimization
- Graph compilation
vllm/compilation/compiler_interface.py - Quantization
csrc/libtorch_stable/quantization
Distributed and Integrations
- Parallel execution
vllm/distributed/device_communicators - Weight transfer
vllm/distributed/weight_transfer/sharded_rdt_engine.py - Metrics and tracing
rust/src/server/src/middleware/metrics.rs
Other components
- API client
Connections 20
- API client → Inference server: submits request
- API client → OpenAI routes: submits request
- OpenAI routes → Chat rendering: renders chat
- Inference server → Engine client: forwards request
- Chat rendering → Tokenization: tokenizes prompt
- Chat rendering → Multimodal inputs: prepares media
- Engine client → Engine core: dispatches request
- LLM engine → Engine core: adds request
- Engine core → Text processing: processes input
- Engine core → Inference kernels: executes model
- Engine core → KV cache: reads and writes
- Engine core → Graph compilation: uses compiled graphs
- Engine core → Quantization: uses quantized kernels
- Engine core → Parallel execution: coordinates workers
- Weight transfer → Engine core: updates weights
- Engine core → Engine client: returns outputs
- Engine client → Inference server: streams results
- Engine core → Metrics and tracing: reports metrics
- Engine client → Stream parsing: parses stream
- Stream parsing → OpenAI routes: formats tool events