# lunaryes/nano-vllm architecture

> Architecture diagram of the GitHub repository lunaryes/nano-vllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lunaryes/nano-vllm
- Repository: https://github.com/lunaryes/nano-vllm

## Overview

Nano-vLLM is a lightweight, offline Qwen3 inference engine. Its user workflow accepts prompts and sampling settings through the LLM API, schedules prefill and decode work, executes model inference, then returns generated token IDs and decoded text. The graph preserves prefix caching, tensor parallelism, and CUDA-graph execution as documented capabilities. Sampled source establishes the API-to-engine and engine-to-runner flow; scheduler and sequence internals are represented at their tree paths, but their detailed wiring was not sampled.

Main entry points: [`nanovllm/engine/model_runner.py`](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/model_runner.py) (Model Runner), [`nanovllm/engine/llm_engine.py`](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/llm_engine.py) (LLM Engine), [`nanovllm/engine/scheduler.py`](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/scheduler.py) (Scheduler), [`nanovllm/llm.py`](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/llm.py) (LLM API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Inference API"]
  node_llm["LLM API<br/>[llm.py]"]
  node_params["Sampling Params<br/>[sampling_params.py]"]
end

subgraph group_runtime["Request Runtime"]
  node_engine["LLM Engine<br/>[llm_engine.py]"]
  node_tokenizer["Tokenizer<br/>[llm_engine.py]"]
  node_scheduler["Scheduler<br/>[scheduler.py]"]
  node_sequence["Sequences<br/>[sequence.py]"]
end

subgraph group_compute["Model Compute"]
  node_runner["Model Runner<br/>[model_runner.py]"]
  node_loader["Weight Loader<br/>[loader.py]"]
  node_qwen["Qwen3 Model<br/>[qwen3.py]"]
  node_layers["Model Layers"]
  node_context["Execution Context<br/>[context.py]"]
  node_sampler["Token Sampler<br/>[sampler.py]"]
end

subgraph group_optimization["Inference Optimizations"]
  node_blocks["KV Block Manager<br/>[block_manager.py]"]
  node_workers["Tensor Parallel Workers<br/>[model_runner.py]"]
end

node_user(("Caller"))
node_weights[("Model Weights")]
node_cuda["CUDA Runtime"]

node_user -->|"submits prompts"| node_llm
node_user -->|"sets sampling"| node_params
node_llm -->|"delegates generation"| node_engine
node_engine -->|"encodes and decodes"| node_tokenizer
node_engine -->|"adds and schedules"| node_scheduler
node_scheduler -->|"manages requests"| node_sequence
node_scheduler -->|"allocates cached blocks"| node_blocks
node_engine -->|"dispatches run"| node_runner
node_runner -->|"loads weights"| node_loader
node_loader -->|"reads tensors"| node_weights
node_runner -->|"runs model"| node_qwen
node_qwen -->|"uses layers"| node_layers
node_runner -->|"sets execution context"| node_context
node_runner -->|"samples logits"| node_sampler
node_engine -->|"applies settings"| node_params
node_runner -->|"executes inference"| node_cuda
node_runner -.->|"dispatches parallel work"| node_workers
node_engine -->|"returns generated text"| node_user

click node_llm "https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/llm.py"
click node_params "https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/sampling_params.py"
click node_engine "https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/llm_engine.py"
click node_tokenizer "https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/llm_engine.py"
click node_scheduler "https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/scheduler.py"
click node_sequence "https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/sequence.py"
click node_blocks "https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/block_manager.py"
click node_runner "https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/model_runner.py"
click node_loader "https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/utils/loader.py"
click node_qwen "https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/models/qwen3.py"
click node_layers "https://github.com/lunaryes/nano-vllm/tree/main/nanovllm/layers"
click node_context "https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/utils/context.py"
click node_sampler "https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/layers/sampler.py"
click node_workers "https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/model_runner.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_llm,node_params toneBlue
class node_engine,node_tokenizer,node_scheduler,node_sequence,node_weights toneAmber
class node_runner,node_loader,node_qwen,node_layers,node_context,node_sampler toneMint
class node_blocks,node_workers toneRose
class node_user,node_cuda toneIndigo
```

## Components

### Inference API

- [LLM API](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/llm.py): `nanovllm/llm.py`
- [Sampling Params](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/sampling_params.py): `nanovllm/sampling_params.py`

### Request Runtime

- [LLM Engine](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/llm_engine.py): `nanovllm/engine/llm_engine.py`
- [Tokenizer](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/llm_engine.py): `nanovllm/engine/llm_engine.py`
- [Scheduler](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/scheduler.py): `nanovllm/engine/scheduler.py`
- [Sequences](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/sequence.py): `nanovllm/engine/sequence.py`

### Model Compute

- [Model Runner](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/model_runner.py): `nanovllm/engine/model_runner.py`
- [Weight Loader](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/utils/loader.py): `nanovllm/utils/loader.py`
- [Qwen3 Model](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/models/qwen3.py): `nanovllm/models/qwen3.py`
- [Model Layers](https://github.com/lunaryes/nano-vllm/tree/main/nanovllm/layers): `nanovllm/layers`
- [Execution Context](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/utils/context.py): `nanovllm/utils/context.py`
- [Token Sampler](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/layers/sampler.py): `nanovllm/layers/sampler.py`

### Inference Optimizations

- [KV Block Manager](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/block_manager.py): `nanovllm/engine/block_manager.py`
- [Tensor Parallel Workers](https://github.com/lunaryes/nano-vllm/blob/main/nanovllm/engine/model_runner.py): `nanovllm/engine/model_runner.py`

### Other components

- **Caller**
- **Model Weights**
- **CUDA Runtime**

## Connections

- Caller → LLM API: submits prompts
- Caller → Sampling Params: sets sampling
- LLM API → LLM Engine: delegates generation
- LLM Engine → Tokenizer: encodes and decodes
- LLM Engine → Scheduler: adds and schedules
- Scheduler → Sequences: manages requests
- Scheduler → KV Block Manager: allocates cached blocks
- LLM Engine → Model Runner: dispatches run
- Model Runner → Weight Loader: loads weights
- Weight Loader → Model Weights: reads tensors
- Model Runner → Qwen3 Model: runs model
- Qwen3 Model → Model Layers: uses layers
- Model Runner → Execution Context: sets execution context
- Model Runner → Token Sampler: samples logits
- LLM Engine → Sampling Params: applies settings
- Model Runner → CUDA Runtime: executes inference
- Model Runner → Tensor Parallel Workers: dispatches parallel work
- LLM Engine → Caller: returns generated text