How vllm-project/vllm works
vLLM is an LLM inference and serving system. A request enters through an API or the Python engine, is configured and rendered into model input, then scheduled and executed across supported hardware; generated output can be streamed and parsed for tool calls or reasoning. The repository also documents broad capabilities—multimodal input, quantization, adapters, distributed execution, and multiple API protocols—whose detailed implementations are not all represented in the sampled excerpts. Edges are limited to README claims and relationships visible in the source index or excerpts.
Main entry points: llm_engine.py LLM Engine · sm100_mla.hpp Optimized Kernels · streaming_parser_engine.py Streaming Parser · routes.rs API Server
Components 22
Serving and APIs
- API Server
rust/src/server/src/routes.rs - gRPC API
rust/src/server/src/grpc/inference.rs - LLM Engine
vllm/v1/engine/llm_engine.py
Request Processing
- Engine Config
vllm/config/__init__.py - Input Prep
vllm/inputs/engine.py - Multimodal Inputs
vllm/multimodal/inputs.py - Streaming Parser
vllm/parser/engine/streaming_parser_engine.py - Parser Engine
vllm/parser/engine/parser_engine.py - Parser Adapters
vllm/parser/engine/registered_adapters.py
Inference Runtime
- Batching and Cache
vllm/v1/core/single_type_kv_cache_manager.py - Distributed Execution
vllm/distributed/parallel_state.py - Optimized Kernels
csrc/attention/mla/cutlass_sm100_mla/device/sm100_mla.hpp - Hardware Backends
vllm/platforms/__init__.py - Model Weights
Model Capabilities
- Quantization
csrc/quantization/marlin/kernel.h - MoE Execution
csrc/moe/moe_ops.h - LoRA Adapters
vllm/lora/model_manager.py - Weight Transfer
vllm/distributed/weight_transfer/sharded_rdt_engine.py
Operations
- Fault Recovery
vllm/v1/fault_tolerance/engine_core_sentinel.py - Metrics
rust/src/metrics/src/api_server.rs - Logging
vllm/logger.py
Other components
- Client
Connections 24
- Client → API Server: sends request
README.md - Client → LLM Engine: submits prompt
README.md - API Server → LLM Engine: serves inference
- gRPC API → LLM Engine: serves inference
README.md - LLM Engine → Engine Config: creates config
vllm/v1/engine/llm_engine.py - LLM Engine → Input Prep: prepares requests
vllm/v1/engine/llm_engine.py - Input Prep → Multimodal Inputs: accepts inputs
vllm/inputs/engine.py - LLM Engine → Batching and Cache: submits work
- Batching and Cache → Distributed Execution: dispatches execution
- Distributed Execution → Hardware Backends: uses hardware
vllm/distributed/parallel_state.py - Hardware Backends → Optimized Kernels: selects kernels
- Optimized Kernels → Model Weights: executes model
README.md - LLM Engine → LoRA Adapters: loads adapters
vllm/v1/engine/llm_engine.py - LLM Engine → Weight Transfer: transfers weights
- LLM Engine → Fault Recovery: recovers failures
- LLM Engine → Metrics: reports metrics
- LLM Engine → Logging: logs activity
vllm/v1/engine/llm_engine.py - LLM Engine → Parser Engine: parses outputs
- Parser Engine → Streaming Parser: orchestrates streaming
vllm/parser/engine/parser_engine.py - Streaming Parser → Parser Adapters: loads adapters
vllm/parser/engine/registered_adapters.py - Streaming Parser → Client: streams results
README.md - Optimized Kernels → Quantization: supports quantization
README.md - Optimized Kernels → MoE Execution: supports MoE
README.md - LLM Engine → Client: returns output
README.md