# tryingtobeingnikhil/vllm_inference_engine architecture

> Architecture diagram of the GitHub repository tryingtobeingnikhil/vllm_inference_engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tryingtobeingnikhil/vllm_inference_engine
- Repository: https://github.com/tryingtobeingnikhil/vllm_inference_engine

## Overview

PageServe is an LLM inference engine with a sequential baseline and a continuous-batching runtime. In the principal workflow, a client submits a generation request; the v2 server queues it, the scheduler plans and applies batch steps, and the model runner executes packed inference before results are streamed back. The runtime also supports paged/prefix KV caching, preemption and CPU swapping, speculative decoding, and metrics. Separate documented client-side workload generation and measurement support benchmarking. The graph relies on README statements and shown source/index references; unsampled capability links are omitted where ownership is not evidenced.

Main entry points: [`inference_engine/engine/scheduler.py`](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/scheduler.py) (Batch scheduler), [`inference_engine/engine/metrics_aggregator.py`](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/metrics_aggregator.py) (Metrics aggregator), [`inference_engine/server/app_v2.py`](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/server/app_v2.py) (Batch API), [`inference_engine/engine/model_runner.py`](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/model_runner.py) (Model runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving APIs"]
  node_v2api["Batch API<br/>[app_v2.py]"]
  node_baselineapi["Baseline API<br/>[app.py]"]
  node_detokenizer["Detokenizer<br/>[detokenizer.py]"]
end

subgraph group_runtime["Inference runtime"]
  node_config["Engine config<br/>[config.py]"]
  node_loader["Model loader<br/>[loader.py]"]
  node_queue["Request queue<br/>[request_queue.py]"]
  node_scheduler["Batch scheduler<br/>[scheduler.py]"]
  node_sequence["Sequence state<br/>[sequence.py]"]
  node_runner["Model runner<br/>[model_runner.py]"]
  node_attention["Paged attention"]
  node_specdecode["Speculative decoding<br/>[spec_decode.py]"]
  node_sequential["Sequential generation<br/>[sequential.py]"]
end

subgraph group_cache["KV cache and memory"]
  node_cacheconfig["Cache sizing<br/>[kv_cache_config.py]"]
  node_allocator["Block allocator<br/>[block_allocator.py]"]
  node_pagedcache[("Paged KV pool<br/>[paged_kv_cache.py]")]
  node_swap[("CPU swap pool")]
  node_kvtracker["KV tracker"]
end

subgraph group_observability["Metrics and measurement"]
  node_stage["Stage tracker<br/>[stage_tracker.py]"]
  node_collector[("Metrics collector<br/>[collector.py]")]
  node_aggregator["Metrics aggregator"]
  node_loadtest["Load-test runner<br/>[runner.py]"]
  node_profiles["Workload profiles<br/>[profiles.py]"]
  node_report["Load reports<br/>[report.py]"]
end

node_client(("API client"))

node_client -->|"submits request"| node_v2api
node_client -->|"submits request"| node_baselineapi
node_v2api -->|"loads settings"| node_config
node_v2api -->|"loads model"| node_loader
node_v2api -->|"enqueues request"| node_queue
node_v2api -->|"starts scheduler"| node_scheduler
node_v2api -->|"uses for output"| node_detokenizer
node_baselineapi -->|"loads model"| node_loader
node_baselineapi -->|"runs generation"| node_sequential
node_baselineapi -->|"records results"| node_collector
node_scheduler -->|"takes requests"| node_queue
node_scheduler -->|"plans sequence work"| node_sequence
node_scheduler -->|"executes step"| node_runner
node_scheduler -->|"requests proposals"| node_specdecode
node_scheduler -->|"allocates blocks"| node_allocator
node_scheduler -->|"swaps sequences"| node_swap
node_scheduler -->|"tracks KV state"| node_kvtracker
node_scheduler -.->|"records stages"| node_stage
node_scheduler -.->|"records results"| node_collector
node_runner -->|"dispatches attention"| node_attention
node_runner -->|"reads and writes"| node_pagedcache
node_runner -->|"uses cache layout"| node_cacheconfig
node_specdecode -->|"verifies proposals"| node_runner
node_pagedcache -->|"uses block tables"| node_allocator
node_swap -->|"claims device blocks"| node_allocator
node_swap -->|"copies KV blocks"| node_pagedcache
node_cacheconfig -.->|"derives from model"| node_loader
node_aggregator -->|"aggregates metrics"| node_collector
node_aggregator -->|"reads stage data"| node_stage
node_aggregator -->|"reads KV stats"| node_kvtracker
node_aggregator -->|"reads pool stats"| node_allocator
node_aggregator -->|"reads pool stats"| node_pagedcache
node_aggregator -->|"reads swap stats"| node_swap
node_aggregator -->|"reads queue stats"| node_queue
node_profiles -->|"supplies workloads"| node_loadtest
node_loadtest -.->|"sends load"| node_client
node_loadtest -->|"produces results"| node_report

click node_v2api "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/server/app_v2.py"
click node_baselineapi "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/server/app.py"
click node_detokenizer "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/server/detokenizer.py"
click node_config "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/config.py"
click node_loader "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/models/loader.py"
click node_queue "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/request_queue.py"
click node_scheduler "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/scheduler.py"
click node_sequence "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/sequence.py"
click node_runner "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/model_runner.py"
click node_attention "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/attention_wrapper.py"
click node_specdecode "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/spec_decode.py"
click node_sequential "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/sequential.py"
click node_cacheconfig "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/kv_cache_config.py"
click node_allocator "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/block_allocator.py"
click node_pagedcache "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/paged_kv_cache.py"
click node_swap "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/cpu_swap_manager.py"
click node_kvtracker "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/kv_cache_tracker.py"
click node_stage "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/stage_tracker.py"
click node_collector "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/metrics/collector.py"
click node_aggregator "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/metrics_aggregator.py"
click node_loadtest "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/load_test/runner.py"
click node_profiles "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/load_test/profiles.py"
click node_report "https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/load_test/report.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_v2api,node_baselineapi,node_detokenizer,node_client toneBlue
class node_config,node_loader,node_queue,node_scheduler,node_sequence,node_runner,node_attention,node_specdecode,node_sequential toneAmber
class node_cacheconfig,node_allocator,node_pagedcache,node_swap,node_kvtracker toneMint
class node_stage,node_collector,node_aggregator,node_loadtest,node_profiles,node_report toneRose
```

## Components

### Serving APIs

- [Batch API](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/server/app_v2.py): `inference_engine/server/app_v2.py`
- [Baseline API](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/server/app.py): `inference_engine/server/app.py`
- [Detokenizer](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/server/detokenizer.py): `inference_engine/server/detokenizer.py`

### Inference runtime

- [Engine config](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/config.py): `inference_engine/config.py`
- [Model loader](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/models/loader.py): `inference_engine/models/loader.py`
- [Request queue](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/request_queue.py): `inference_engine/engine/request_queue.py`
- [Batch scheduler](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/scheduler.py): `inference_engine/engine/scheduler.py`
- [Sequence state](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/sequence.py): `inference_engine/engine/sequence.py`
- [Model runner](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/model_runner.py): `inference_engine/engine/model_runner.py`
- [Paged attention](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/attention_wrapper.py): `inference_engine/engine/attention_wrapper.py`
- [Speculative decoding](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/spec_decode.py): `inference_engine/engine/spec_decode.py`
- [Sequential generation](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/sequential.py): `inference_engine/engine/sequential.py`

### KV cache and memory

- [Cache sizing](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/kv_cache_config.py): `inference_engine/engine/kv_cache_config.py`
- [Block allocator](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/block_allocator.py): `inference_engine/engine/block_allocator.py`
- [Paged KV pool](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/paged_kv_cache.py): `inference_engine/engine/paged_kv_cache.py`
- [CPU swap pool](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/cpu_swap_manager.py): `inference_engine/engine/cpu_swap_manager.py`
- [KV tracker](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/kv_cache_tracker.py): `inference_engine/engine/kv_cache_tracker.py`

### Metrics and measurement

- [Stage tracker](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/stage_tracker.py): `inference_engine/engine/stage_tracker.py`
- [Metrics collector](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/metrics/collector.py): `inference_engine/metrics/collector.py`
- [Metrics aggregator](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/inference_engine/engine/metrics_aggregator.py): `inference_engine/engine/metrics_aggregator.py`
- [Load-test runner](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/load_test/runner.py): `load_test/runner.py`
- [Workload profiles](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/load_test/profiles.py): `load_test/profiles.py`
- [Load reports](https://github.com/tryingtobeingnikhil/vllm_inference_engine/blob/main/load_test/report.py): `load_test/report.py`

### Other components

- **API client**

## Connections

- API client → Batch API: submits request
- API client → Baseline API: submits request
- Batch API → Engine config: loads settings (evidence: `inference_engine/server/app_v2.py`)
- Batch API → Model loader: loads model (evidence: `inference_engine/server/app_v2.py`)
- Batch API → Request queue: enqueues request (evidence: `inference_engine/server/app_v2.py`)
- Batch API → Batch scheduler: starts scheduler (evidence: `inference_engine/server/app_v2.py`)
- Batch API → Detokenizer: uses for output (evidence: `inference_engine/server/app_v2.py`)
- Baseline API → Model loader: loads model (evidence: `inference_engine/server/app.py`)
- Baseline API → Sequential generation: runs generation (evidence: `inference_engine/server/app.py`)
- Baseline API → Metrics collector: records results (evidence: `inference_engine/server/app.py`)
- Batch scheduler → Request queue: takes requests (evidence: `inference_engine/engine/scheduler.py`)
- Batch scheduler → Sequence state: plans sequence work (evidence: `inference_engine/engine/scheduler.py`)
- Batch scheduler → Model runner: executes step (evidence: `inference_engine/engine/scheduler.py`)
- Batch scheduler → Speculative decoding: requests proposals (evidence: `inference_engine/engine/scheduler.py`)
- Batch scheduler → Block allocator: allocates blocks (evidence: `inference_engine/engine/scheduler.py`)
- Batch scheduler → CPU swap pool: swaps sequences (evidence: `inference_engine/engine/scheduler.py`)
- Batch scheduler → KV tracker: tracks KV state (evidence: `inference_engine/engine/scheduler.py`)
- Batch scheduler → Stage tracker: records stages (evidence: `inference_engine/engine/scheduler.py`)
- Batch scheduler → Metrics collector: records results (evidence: `inference_engine/engine/scheduler.py`)
- Model runner → Paged attention: dispatches attention (evidence: `inference_engine/engine/model_runner.py`)
- Model runner → Paged KV pool: reads and writes (evidence: `inference_engine/engine/model_runner.py`)
- Model runner → Cache sizing: uses cache layout (evidence: `inference_engine/engine/model_runner.py`)
- Speculative decoding → Model runner: verifies proposals (evidence: `inference_engine/engine/spec_decode.py`)
- Paged KV pool → Block allocator: uses block tables (evidence: `inference_engine/engine/paged_kv_cache.py`)
- CPU swap pool → Block allocator: claims device blocks (evidence: `inference_engine/engine/cpu_swap_manager.py`)
- CPU swap pool → Paged KV pool: copies KV blocks (evidence: `inference_engine/engine/cpu_swap_manager.py`)
- Cache sizing → Model loader: derives from model
- Metrics aggregator → Metrics collector: aggregates metrics (evidence: `inference_engine/engine/metrics_aggregator.py`)
- Metrics aggregator → Stage tracker: reads stage data (evidence: `inference_engine/engine/metrics_aggregator.py`)
- Metrics aggregator → KV tracker: reads KV stats (evidence: `inference_engine/engine/metrics_aggregator.py`)
- Metrics aggregator → Block allocator: reads pool stats (evidence: `inference_engine/engine/metrics_aggregator.py`)
- Metrics aggregator → Paged KV pool: reads pool stats (evidence: `inference_engine/engine/metrics_aggregator.py`)
- Metrics aggregator → CPU swap pool: reads swap stats (evidence: `inference_engine/engine/metrics_aggregator.py`)
- Metrics aggregator → Request queue: reads queue stats (evidence: `inference_engine/engine/metrics_aggregator.py`)
- Workload profiles → Load-test runner: supplies workloads (evidence: `load_test/runner.py`)
- Load-test runner → API client: sends load
- Load-test runner → Load reports: produces results (evidence: `load_test/report.py`)
