# vllm-project/vllm architecture

> Architecture diagram of the GitHub repository vllm-project/vllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vllm-project/vllm
- Repository: https://github.com/vllm-project/vllm
- Video tour (about a minute): https://gitdiagram.com/vllm-project/vllm/video

## Overview

vLLM is an LLM inference and serving system. A request enters through an API or the Python engine, is configured and rendered into model input, then scheduled and executed across supported hardware; generated output can be streamed and parsed for tool calls or reasoning. The repository also documents broad capabilities—multimodal input, quantization, adapters, distributed execution, and multiple API protocols—whose detailed implementations are not all represented in the sampled excerpts. Edges are limited to README claims and relationships visible in the source index or excerpts.

Main entry points: [`vllm/v1/engine/llm_engine.py`](https://github.com/vllm-project/vllm/blob/main/vllm/v1/engine/llm_engine.py) (LLM Engine), [`csrc/attention/mla/cutlass_sm100_mla/device/sm100_mla.hpp`](https://github.com/vllm-project/vllm/blob/main/csrc/attention/mla/cutlass_sm100_mla/device/sm100_mla.hpp) (Optimized Kernels), [`vllm/parser/engine/streaming_parser_engine.py`](https://github.com/vllm-project/vllm/blob/main/vllm/parser/engine/streaming_parser_engine.py) (Streaming Parser), [`rust/src/server/src/routes.rs`](https://github.com/vllm-project/vllm/blob/main/rust/src/server/src/routes.rs) (API Server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving and APIs"]
  node_api["API Server<br/>[routes.rs]"]
  node_grpc["gRPC API<br/>[inference.rs]"]
  node_engine["LLM Engine<br/>[llm_engine.py]"]
end

subgraph group_request["Request Processing"]
  node_config["Engine Config<br/>[__init__.py]"]
  node_input["Input Prep<br/>[engine.py]"]
  node_multimodal["Multimodal Inputs<br/>[inputs.py]"]
  node_parser["Streaming Parser"]
  node_parser_engine["Parser Engine<br/>[parser_engine.py]"]
  node_adapters["Parser Adapters"]
end

subgraph group_runtime["Inference Runtime"]
  node_scheduler["Batching and Cache"]
  node_parallel["Distributed Execution<br/>[parallel_state.py]"]
  node_kernels["Optimized Kernels<br/>[sm100_mla.hpp]"]
  node_platform["Hardware Backends<br/>[__init__.py]"]
  node_models["Model Weights"]
end

subgraph group_extensions["Model Capabilities"]
  node_quant["Quantization<br/>[kernel.h]"]
  node_moe["MoE Execution<br/>[moe_ops.h]"]
  node_lora["LoRA Adapters<br/>[model_manager.py]"]
  node_weight_transfer["Weight Transfer"]
end

subgraph group_ops["Operations"]
  node_sentinel["Fault Recovery"]
  node_metrics["Metrics<br/>[api_server.rs]"]
  node_logs["Logging<br/>[logger.py]"]
end

node_user(("Client"))

node_user -->|"sends request"| node_api
node_user -->|"submits prompt"| node_engine
node_api -.->|"serves inference"| node_engine
node_grpc -.->|"serves inference"| node_engine
node_engine -->|"creates config"| node_config
node_engine -->|"prepares requests"| node_input
node_input -->|"accepts inputs"| node_multimodal
node_engine -.->|"submits work"| node_scheduler
node_scheduler -.->|"dispatches execution"| node_parallel
node_parallel -.->|"uses hardware"| node_platform
node_platform -.->|"selects kernels"| node_kernels
node_kernels -.->|"executes model"| node_models
node_engine -->|"loads adapters"| node_lora
node_engine -.->|"transfers weights"| node_weight_transfer
node_engine -.->|"recovers failures"| node_sentinel
node_engine -.->|"reports metrics"| node_metrics
node_engine -.->|"logs activity"| node_logs
node_engine -.->|"parses outputs"| node_parser_engine
node_parser_engine -->|"orchestrates streaming"| node_parser
node_parser -->|"loads adapters"| node_adapters
node_parser -.->|"streams results"| node_user
node_kernels -.->|"supports quantization"| node_quant
node_kernels -.->|"supports MoE"| node_moe
node_engine -.->|"returns output"| node_user

click node_api "https://github.com/vllm-project/vllm/blob/main/rust/src/server/src/routes.rs"
click node_grpc "https://github.com/vllm-project/vllm/blob/main/rust/src/server/src/grpc/inference.rs"
click node_engine "https://github.com/vllm-project/vllm/blob/main/vllm/v1/engine/llm_engine.py"
click node_config "https://github.com/vllm-project/vllm/blob/main/vllm/config/__init__.py"
click node_input "https://github.com/vllm-project/vllm/blob/main/vllm/inputs/engine.py"
click node_multimodal "https://github.com/vllm-project/vllm/blob/main/vllm/multimodal/inputs.py"
click node_parser "https://github.com/vllm-project/vllm/blob/main/vllm/parser/engine/streaming_parser_engine.py"
click node_parser_engine "https://github.com/vllm-project/vllm/blob/main/vllm/parser/engine/parser_engine.py"
click node_adapters "https://github.com/vllm-project/vllm/blob/main/vllm/parser/engine/registered_adapters.py"
click node_scheduler "https://github.com/vllm-project/vllm/blob/main/vllm/v1/core/single_type_kv_cache_manager.py"
click node_parallel "https://github.com/vllm-project/vllm/blob/main/vllm/distributed/parallel_state.py"
click node_kernels "https://github.com/vllm-project/vllm/blob/main/csrc/attention/mla/cutlass_sm100_mla/device/sm100_mla.hpp"
click node_platform "https://github.com/vllm-project/vllm/blob/main/vllm/platforms/__init__.py"
click node_quant "https://github.com/vllm-project/vllm/blob/main/csrc/quantization/marlin/kernel.h"
click node_moe "https://github.com/vllm-project/vllm/blob/main/csrc/moe/moe_ops.h"
click node_lora "https://github.com/vllm-project/vllm/blob/main/vllm/lora/model_manager.py"
click node_weight_transfer "https://github.com/vllm-project/vllm/blob/main/vllm/distributed/weight_transfer/sharded_rdt_engine.py"
click node_sentinel "https://github.com/vllm-project/vllm/blob/main/vllm/v1/fault_tolerance/engine_core_sentinel.py"
click node_metrics "https://github.com/vllm-project/vllm/blob/main/rust/src/metrics/src/api_server.rs"
click node_logs "https://github.com/vllm-project/vllm/blob/main/vllm/logger.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_grpc,node_engine,node_user toneBlue
class node_config,node_input,node_multimodal,node_parser,node_parser_engine,node_adapters toneAmber
class node_scheduler,node_parallel,node_kernels,node_platform,node_models toneMint
class node_quant,node_moe,node_lora,node_weight_transfer toneRose
class node_sentinel,node_metrics,node_logs toneIndigo
```

## Components

### Serving and APIs

- [API Server](https://github.com/vllm-project/vllm/blob/main/rust/src/server/src/routes.rs): `rust/src/server/src/routes.rs`
- [gRPC API](https://github.com/vllm-project/vllm/blob/main/rust/src/server/src/grpc/inference.rs): `rust/src/server/src/grpc/inference.rs`
- [LLM Engine](https://github.com/vllm-project/vllm/blob/main/vllm/v1/engine/llm_engine.py): `vllm/v1/engine/llm_engine.py`

### Request Processing

- [Engine Config](https://github.com/vllm-project/vllm/blob/main/vllm/config/__init__.py): `vllm/config/__init__.py`
- [Input Prep](https://github.com/vllm-project/vllm/blob/main/vllm/inputs/engine.py): `vllm/inputs/engine.py`
- [Multimodal Inputs](https://github.com/vllm-project/vllm/blob/main/vllm/multimodal/inputs.py): `vllm/multimodal/inputs.py`
- [Streaming Parser](https://github.com/vllm-project/vllm/blob/main/vllm/parser/engine/streaming_parser_engine.py): `vllm/parser/engine/streaming_parser_engine.py`
- [Parser Engine](https://github.com/vllm-project/vllm/blob/main/vllm/parser/engine/parser_engine.py): `vllm/parser/engine/parser_engine.py`
- [Parser Adapters](https://github.com/vllm-project/vllm/blob/main/vllm/parser/engine/registered_adapters.py): `vllm/parser/engine/registered_adapters.py`

### Inference Runtime

- [Batching and Cache](https://github.com/vllm-project/vllm/blob/main/vllm/v1/core/single_type_kv_cache_manager.py): `vllm/v1/core/single_type_kv_cache_manager.py`
- [Distributed Execution](https://github.com/vllm-project/vllm/blob/main/vllm/distributed/parallel_state.py): `vllm/distributed/parallel_state.py`
- [Optimized Kernels](https://github.com/vllm-project/vllm/blob/main/csrc/attention/mla/cutlass_sm100_mla/device/sm100_mla.hpp): `csrc/attention/mla/cutlass_sm100_mla/device/sm100_mla.hpp`
- [Hardware Backends](https://github.com/vllm-project/vllm/blob/main/vllm/platforms/__init__.py): `vllm/platforms/__init__.py`
- **Model Weights**

### Model Capabilities

- [Quantization](https://github.com/vllm-project/vllm/blob/main/csrc/quantization/marlin/kernel.h): `csrc/quantization/marlin/kernel.h`
- [MoE Execution](https://github.com/vllm-project/vllm/blob/main/csrc/moe/moe_ops.h): `csrc/moe/moe_ops.h`
- [LoRA Adapters](https://github.com/vllm-project/vllm/blob/main/vllm/lora/model_manager.py): `vllm/lora/model_manager.py`
- [Weight Transfer](https://github.com/vllm-project/vllm/blob/main/vllm/distributed/weight_transfer/sharded_rdt_engine.py): `vllm/distributed/weight_transfer/sharded_rdt_engine.py`

### Operations

- [Fault Recovery](https://github.com/vllm-project/vllm/blob/main/vllm/v1/fault_tolerance/engine_core_sentinel.py): `vllm/v1/fault_tolerance/engine_core_sentinel.py`
- [Metrics](https://github.com/vllm-project/vllm/blob/main/rust/src/metrics/src/api_server.rs): `rust/src/metrics/src/api_server.rs`
- [Logging](https://github.com/vllm-project/vllm/blob/main/vllm/logger.py): `vllm/logger.py`

### Other components

- **Client**

## Connections

- Client → API Server: sends request (evidence: `README.md`)
- Client → LLM Engine: submits prompt (evidence: `README.md`)
- API Server → LLM Engine: serves inference
- gRPC API → LLM Engine: serves inference (evidence: `README.md`)
- LLM Engine → Engine Config: creates config (evidence: `vllm/v1/engine/llm_engine.py`)
- LLM Engine → Input Prep: prepares requests (evidence: `vllm/v1/engine/llm_engine.py`)
- Input Prep → Multimodal Inputs: accepts inputs (evidence: `vllm/inputs/engine.py`)
- LLM Engine → Batching and Cache: submits work
- Batching and Cache → Distributed Execution: dispatches execution
- Distributed Execution → Hardware Backends: uses hardware (evidence: `vllm/distributed/parallel_state.py`)
- Hardware Backends → Optimized Kernels: selects kernels
- Optimized Kernels → Model Weights: executes model (evidence: `README.md`)
- LLM Engine → LoRA Adapters: loads adapters (evidence: `vllm/v1/engine/llm_engine.py`)
- LLM Engine → Weight Transfer: transfers weights
- LLM Engine → Fault Recovery: recovers failures
- LLM Engine → Metrics: reports metrics
- LLM Engine → Logging: logs activity (evidence: `vllm/v1/engine/llm_engine.py`)
- LLM Engine → Parser Engine: parses outputs
- Parser Engine → Streaming Parser: orchestrates streaming (evidence: `vllm/parser/engine/parser_engine.py`)
- Streaming Parser → Parser Adapters: loads adapters (evidence: `vllm/parser/engine/registered_adapters.py`)
- Streaming Parser → Client: streams results (evidence: `README.md`)
- Optimized Kernels → Quantization: supports quantization (evidence: `README.md`)
- Optimized Kernels → MoE Execution: supports MoE (evidence: `README.md`)
- LLM Engine → Client: returns output (evidence: `README.md`)
