How tryingtobeingnikhil/vllm_inference_engine works
PageServe is an LLM inference engine with a sequential baseline and a continuous-batching runtime. In the principal workflow, a client submits a generation request; the v2 server queues it, the scheduler plans and applies batch steps, and the model runner executes packed inference before results are streamed back. The runtime also supports paged/prefix KV caching, preemption and CPU swapping, speculative decoding, and metrics. Separate documented client-side workload generation and measurement support benchmarking. The graph relies on README statements and shown source/index references; unsampled capability links are omitted where ownership is not evidenced.
Main entry points: scheduler.py Batch scheduler · metrics_aggregator.py Metrics aggregator · app_v2.py Batch API · model_runner.py Model runner
Components 24
Serving APIs
- Batch API
inference_engine/server/app_v2.py - Baseline API
inference_engine/server/app.py - Detokenizer
inference_engine/server/detokenizer.py
Inference runtime
- Engine config
inference_engine/config.py - Model loader
inference_engine/models/loader.py - Request queue
inference_engine/engine/request_queue.py - Batch scheduler
inference_engine/engine/scheduler.py - Sequence state
inference_engine/engine/sequence.py - Model runner
inference_engine/engine/model_runner.py - Paged attention
inference_engine/engine/attention_wrapper.py - Speculative decoding
inference_engine/engine/spec_decode.py - Sequential generation
inference_engine/engine/sequential.py
KV cache and memory
- Cache sizing
inference_engine/engine/kv_cache_config.py - Block allocator
inference_engine/engine/block_allocator.py - Paged KV pool
inference_engine/engine/paged_kv_cache.py - CPU swap pool
inference_engine/engine/cpu_swap_manager.py - KV tracker
inference_engine/engine/kv_cache_tracker.py
Metrics and measurement
- Stage tracker
inference_engine/engine/stage_tracker.py - Metrics collector
inference_engine/metrics/collector.py - Metrics aggregator
inference_engine/engine/metrics_aggregator.py - Load-test runner
load_test/runner.py - Workload profiles
load_test/profiles.py - Load reports
load_test/report.py
Other components
- API client
Connections 37
- API client → Batch API: submits request
- API client → Baseline API: submits request
- Batch API → Engine config: loads settings
inference_engine/server/app_v2.py - Batch API → Model loader: loads model
inference_engine/server/app_v2.py - Batch API → Request queue: enqueues request
inference_engine/server/app_v2.py - Batch API → Batch scheduler: starts scheduler
inference_engine/server/app_v2.py - Batch API → Detokenizer: uses for output
inference_engine/server/app_v2.py - Baseline API → Model loader: loads model
inference_engine/server/app.py - Baseline API → Sequential generation: runs generation
inference_engine/server/app.py - Baseline API → Metrics collector: records results
inference_engine/server/app.py - Batch scheduler → Request queue: takes requests
inference_engine/engine/scheduler.py - Batch scheduler → Sequence state: plans sequence work
inference_engine/engine/scheduler.py - Batch scheduler → Model runner: executes step
inference_engine/engine/scheduler.py - Batch scheduler → Speculative decoding: requests proposals
inference_engine/engine/scheduler.py - Batch scheduler → Block allocator: allocates blocks
inference_engine/engine/scheduler.py - Batch scheduler → CPU swap pool: swaps sequences
inference_engine/engine/scheduler.py - Batch scheduler → KV tracker: tracks KV state
inference_engine/engine/scheduler.py - Batch scheduler → Stage tracker: records stages
inference_engine/engine/scheduler.py - Batch scheduler → Metrics collector: records results
inference_engine/engine/scheduler.py - Model runner → Paged attention: dispatches attention
inference_engine/engine/model_runner.py - Model runner → Paged KV pool: reads and writes
inference_engine/engine/model_runner.py - Model runner → Cache sizing: uses cache layout
inference_engine/engine/model_runner.py - Speculative decoding → Model runner: verifies proposals
inference_engine/engine/spec_decode.py - Paged KV pool → Block allocator: uses block tables
inference_engine/engine/paged_kv_cache.py - CPU swap pool → Block allocator: claims device blocks
inference_engine/engine/cpu_swap_manager.py - CPU swap pool → Paged KV pool: copies KV blocks
inference_engine/engine/cpu_swap_manager.py - Cache sizing → Model loader: derives from model
- Metrics aggregator → Metrics collector: aggregates metrics
inference_engine/engine/metrics_aggregator.py - Metrics aggregator → Stage tracker: reads stage data
inference_engine/engine/metrics_aggregator.py - Metrics aggregator → KV tracker: reads KV stats
inference_engine/engine/metrics_aggregator.py - Metrics aggregator → Block allocator: reads pool stats
inference_engine/engine/metrics_aggregator.py - Metrics aggregator → Paged KV pool: reads pool stats
inference_engine/engine/metrics_aggregator.py - Metrics aggregator → CPU swap pool: reads swap stats
inference_engine/engine/metrics_aggregator.py - Metrics aggregator → Request queue: reads queue stats
inference_engine/engine/metrics_aggregator.py - Workload profiles → Load-test runner: supplies workloads
load_test/runner.py - Load-test runner → API client: sends load
- Load-test runner → Load reports: produces results
load_test/report.py