How lunaryes/nano-vllm works
Nano-vLLM is a lightweight, offline Qwen3 inference engine. Its user workflow accepts prompts and sampling settings through the LLM API, schedules prefill and decode work, executes model inference, then returns generated token IDs and decoded text. The graph preserves prefix caching, tensor parallelism, and CUDA-graph execution as documented capabilities. Sampled source establishes the API-to-engine and engine-to-runner flow; scheduler and sequence internals are represented at their tree paths, but their detailed wiring was not sampled.
Main entry points: model_runner.py Model Runner · llm_engine.py LLM Engine · scheduler.py Scheduler · llm.py LLM API
Components 17
Inference API
- LLM API
nanovllm/llm.py - Sampling Params
nanovllm/sampling_params.py
Request Runtime
- LLM Engine
nanovllm/engine/llm_engine.py - Tokenizer
nanovllm/engine/llm_engine.py - Scheduler
nanovllm/engine/scheduler.py - Sequences
nanovllm/engine/sequence.py
Model Compute
- Model Runner
nanovllm/engine/model_runner.py - Weight Loader
nanovllm/utils/loader.py - Qwen3 Model
nanovllm/models/qwen3.py - Model Layers
nanovllm/layers - Execution Context
nanovllm/utils/context.py - Token Sampler
nanovllm/layers/sampler.py
Inference Optimizations
- KV Block Manager
nanovllm/engine/block_manager.py - Tensor Parallel Workers
nanovllm/engine/model_runner.py
Other components
- Caller
- Model Weights
- CUDA Runtime
Connections 18
- Caller → LLM API: submits prompts
- Caller → Sampling Params: sets sampling
- LLM API → LLM Engine: delegates generation
- LLM Engine → Tokenizer: encodes and decodes
- LLM Engine → Scheduler: adds and schedules
- Scheduler → Sequences: manages requests
- Scheduler → KV Block Manager: allocates cached blocks
- LLM Engine → Model Runner: dispatches run
- Model Runner → Weight Loader: loads weights
- Weight Loader → Model Weights: reads tensors
- Model Runner → Qwen3 Model: runs model
- Qwen3 Model → Model Layers: uses layers
- Model Runner → Execution Context: sets execution context
- Model Runner → Token Sampler: samples logits
- LLM Engine → Sampling Params: applies settings
- Model Runner → CUDA Runtime: executes inference
- Model Runner → Tensor Parallel Workers: dispatches parallel work
- LLM Engine → Caller: returns generated text