# nopnoping/micro-vllm architecture

> Architecture diagram of the GitHub repository nopnoping/micro-vllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nopnoping/micro-vllm
- Repository: https://github.com/nopnoping/micro-vllm

## Overview

micro-vLLM is a lightweight, synchronous inference engine for decoder-only language models. The principal workflow accepts prompts through the public LLM API, tokenizes them, schedules requests under token and sequence budgets, manages logical KV-cache pages, executes immutable plans through an executor and worker, then samples and decodes generated tokens. The map also preserves model loading, supported architectures, replaceable attention backends, and tracing. Relationships are limited to README statements and shown source references/calls; some internal implementations were not excerpted, so uncertain links are omitted.

Main entry points: [`src/micro_vllm/llm.py`](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/llm.py) (LLM API), [`src/micro_vllm/engine/engine.py`](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/engine/engine.py) (Engine), [`src/micro_vllm/attention/attention.py`](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/attention/attention.py) (Attention Runtime), [`src/micro_vllm/core/scheduler.py`](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/scheduler.py) (Scheduler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Public API"]
  node_llm["LLM API<br/>[llm.py]"]
  node_tokenizer["Tokenizer<br/>[tokenizer.py]"]
  node_sampling_params["Sampling Parameters<br/>[params.py]"]
  node_output["Generation Output<br/>[output.py]"]
end

subgraph group_control["Request Control"]
  node_engine["Engine<br/>[engine.py]"]
  node_scheduler["Scheduler<br/>[scheduler.py]"]
  node_request["Request State<br/>[request.py]"]
  node_plan["Execution Plan"]
  node_cache["KV Cache Manager"]
  node_cache_groups["Cache Groups"]
  node_blocks[("Cache Blocks<br/>[block.py]")]
  node_prefix[("Prefix Cache<br/>[prefix_cache.py]")]
end

subgraph group_runtime["Model Runtime"]
  node_executor["Executor<br/>[single_process.py]"]
  node_worker["Worker<br/>[worker.py]"]
  node_runner["Model Runner<br/>[model_runner.py]"]
  node_loader["Model Loader<br/>[default.py]"]
  node_sampler["Sampler<br/>[sampler.py]"]
end

subgraph group_model["Models and Operators"]
  node_model_registry["Model Registry<br/>[registry.py]"]
  node_llama["Llama<br/>[llama.py]"]
  node_mistral["Mistral<br/>[mistral.py]"]
  node_qwen["Qwen3<br/>[qwen.py]"]
  node_layers["Model Layers"]
  node_attention["Attention Runtime<br/>[attention.py]"]
  node_attention_spec["Attention Semantics<br/>[spec.py]"]
  node_operators["Attention Operators<br/>[registry.py]"]
end

subgraph group_support["Configuration and Trace"]
  node_config["Engine Configuration<br/>[config.py]"]
  node_trace["Trace Sink<br/>[trace.py]"]
end

node_user(("Caller"))

node_user -->|"submits prompts"| node_llm
node_llm -->|"tokenizes prompts"| node_tokenizer
node_llm -->|"runs generation"| node_engine
node_llm -->|"sets policy"| node_sampling_params
node_engine -->|"schedules requests"| node_scheduler
node_engine -->|"creates requests"| node_request
node_engine -->|"executes plan"| node_executor
node_engine -->|"builds outputs"| node_output
node_scheduler -->|"manages cache"| node_cache
node_scheduler -->|"updates state"| node_request
node_scheduler -->|"produces plan"| node_plan
node_cache -->|"coordinates groups"| node_cache_groups
node_cache_groups -->|"allocates blocks"| node_blocks
node_cache_groups -->|"uses prefix cache"| node_prefix
node_executor -->|"dispatches plan"| node_worker
node_worker -->|"delegates execution"| node_runner
node_runner -->|"loads model"| node_loader
node_loader -->|"selects architecture"| node_model_registry
node_model_registry -->|"resolves model"| node_llama
node_model_registry -->|"resolves model"| node_mistral
node_model_registry -->|"resolves model"| node_qwen
node_llama -->|"uses layers"| node_layers
node_mistral -->|"uses layers"| node_layers
node_qwen -->|"uses layers"| node_layers
node_llama -->|"runs attention"| node_attention
node_mistral -->|"runs attention"| node_attention
node_qwen -->|"runs attention"| node_attention
node_attention -->|"builds visibility"| node_attention_spec
node_attention -->|"selects operator"| node_operators
node_runner -->|"samples logits"| node_sampler
node_sampler -->|"applies policy"| node_sampling_params
node_llm -->|"configures engine"| node_config
node_runner -->|"reads settings"| node_config
node_config -->|"accepts trace sink"| node_trace

click node_llm "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/llm.py"
click node_tokenizer "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/tokenizer.py"
click node_sampling_params "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/sampling/params.py"
click node_engine "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/engine/engine.py"
click node_scheduler "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/scheduler.py"
click node_request "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/request.py"
click node_plan "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/scheduler_output.py"
click node_cache "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/kv_cache_manager.py"
click node_cache_groups "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/cache_group_manager.py"
click node_blocks "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/block.py"
click node_prefix "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/prefix_cache.py"
click node_executor "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/executor/single_process.py"
click node_worker "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/worker/worker.py"
click node_runner "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/worker/model_runner.py"
click node_loader "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/model_loader/default.py"
click node_model_registry "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/models/registry.py"
click node_llama "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/models/llama.py"
click node_mistral "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/models/mistral.py"
click node_qwen "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/models/qwen.py"
click node_layers "https://github.com/nopnoping/micro-vllm/tree/master/src/micro_vllm/layers"
click node_attention "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/attention/attention.py"
click node_attention_spec "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/attention/spec.py"
click node_operators "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/attention/operators/registry.py"
click node_sampler "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/sampling/sampler.py"
click node_output "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/engine/output.py"
click node_config "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/config.py"
click node_trace "https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/trace.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_llm,node_tokenizer,node_sampling_params,node_output toneBlue
class node_engine,node_scheduler,node_request,node_plan,node_cache,node_cache_groups,node_blocks,node_prefix toneAmber
class node_executor,node_worker,node_runner,node_loader,node_sampler toneMint
class node_model_registry,node_llama,node_mistral,node_qwen,node_layers,node_attention,node_attention_spec,node_operators toneRose
class node_config,node_trace,node_user toneIndigo
```

## Components

### Public API

- [LLM API](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/llm.py): `src/micro_vllm/llm.py`
- [Tokenizer](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/tokenizer.py): `src/micro_vllm/tokenizer.py`
- [Sampling Parameters](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/sampling/params.py): `src/micro_vllm/sampling/params.py`
- [Generation Output](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/engine/output.py): `src/micro_vllm/engine/output.py`

### Request Control

- [Engine](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/engine/engine.py): `src/micro_vllm/engine/engine.py`
- [Scheduler](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/scheduler.py): `src/micro_vllm/core/scheduler.py`
- [Request State](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/request.py): `src/micro_vllm/core/request.py`
- [Execution Plan](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/scheduler_output.py): `src/micro_vllm/core/scheduler_output.py`
- [KV Cache Manager](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/kv_cache_manager.py): `src/micro_vllm/core/kv_cache_manager.py`
- [Cache Groups](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/cache_group_manager.py): `src/micro_vllm/core/cache_group_manager.py`
- [Cache Blocks](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/block.py): `src/micro_vllm/core/block.py`
- [Prefix Cache](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/core/prefix_cache.py): `src/micro_vllm/core/prefix_cache.py`

### Model Runtime

- [Executor](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/executor/single_process.py): `src/micro_vllm/executor/single_process.py`
- [Worker](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/worker/worker.py): `src/micro_vllm/worker/worker.py`
- [Model Runner](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/worker/model_runner.py): `src/micro_vllm/worker/model_runner.py`
- [Model Loader](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/model_loader/default.py): `src/micro_vllm/model_loader/default.py`
- [Sampler](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/sampling/sampler.py): `src/micro_vllm/sampling/sampler.py`

### Models and Operators

- [Model Registry](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/models/registry.py): `src/micro_vllm/models/registry.py`
- [Llama](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/models/llama.py): `src/micro_vllm/models/llama.py`
- [Mistral](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/models/mistral.py): `src/micro_vllm/models/mistral.py`
- [Qwen3](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/models/qwen.py): `src/micro_vllm/models/qwen.py`
- [Model Layers](https://github.com/nopnoping/micro-vllm/tree/master/src/micro_vllm/layers): `src/micro_vllm/layers`
- [Attention Runtime](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/attention/attention.py): `src/micro_vllm/attention/attention.py`
- [Attention Semantics](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/attention/spec.py): `src/micro_vllm/attention/spec.py`
- [Attention Operators](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/attention/operators/registry.py): `src/micro_vllm/attention/operators/registry.py`

### Configuration and Trace

- [Engine Configuration](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/config.py): `src/micro_vllm/config.py`
- [Trace Sink](https://github.com/nopnoping/micro-vllm/blob/master/src/micro_vllm/trace.py): `src/micro_vllm/trace.py`

### Other components

- **Caller**

## Connections

- Caller → LLM API: submits prompts
- LLM API → Tokenizer: tokenizes prompts (evidence: `src/micro_vllm/llm.py`)
- LLM API → Engine: runs generation (evidence: `src/micro_vllm/llm.py`)
- LLM API → Sampling Parameters: sets policy (evidence: `src/micro_vllm/llm.py`)
- Engine → Scheduler: schedules requests (evidence: `src/micro_vllm/engine/engine.py`)
- Engine → Request State: creates requests (evidence: `src/micro_vllm/engine/engine.py`)
- Engine → Executor: executes plan (evidence: `src/micro_vllm/engine/engine.py`)
- Engine → Generation Output: builds outputs (evidence: `src/micro_vllm/engine/engine.py`)
- Scheduler → KV Cache Manager: manages cache (evidence: `src/micro_vllm/core/scheduler.py`)
- Scheduler → Request State: updates state (evidence: `src/micro_vllm/core/scheduler.py`)
- Scheduler → Execution Plan: produces plan (evidence: `src/micro_vllm/core/scheduler.py`)
- KV Cache Manager → Cache Groups: coordinates groups (evidence: `src/micro_vllm/core/kv_cache_manager.py`)
- Cache Groups → Cache Blocks: allocates blocks (evidence: `src/micro_vllm/core/cache_group_manager.py`)
- Cache Groups → Prefix Cache: uses prefix cache (evidence: `src/micro_vllm/core/cache_group_manager.py`)
- Executor → Worker: dispatches plan (evidence: `src/micro_vllm/executor/single_process.py`)
- Worker → Model Runner: delegates execution (evidence: `src/micro_vllm/worker/worker.py`)
- Model Runner → Model Loader: loads model (evidence: `src/micro_vllm/worker/model_runner.py`)
- Model Loader → Model Registry: selects architecture
- Model Registry → Llama: resolves model (evidence: `src/micro_vllm/models/registry.py`)
- Model Registry → Mistral: resolves model (evidence: `src/micro_vllm/models/registry.py`)
- Model Registry → Qwen3: resolves model (evidence: `src/micro_vllm/models/registry.py`)
- Llama → Model Layers: uses layers (evidence: `src/micro_vllm/models/llama.py`)
- Mistral → Model Layers: uses layers (evidence: `src/micro_vllm/models/mistral.py`)
- Qwen3 → Model Layers: uses layers (evidence: `src/micro_vllm/models/qwen.py`)
- Llama → Attention Runtime: runs attention (evidence: `src/micro_vllm/models/llama.py`)
- Mistral → Attention Runtime: runs attention (evidence: `src/micro_vllm/models/mistral.py`)
- Qwen3 → Attention Runtime: runs attention (evidence: `src/micro_vllm/models/qwen.py`)
- Attention Runtime → Attention Semantics: builds visibility (evidence: `src/micro_vllm/attention/attention.py`)
- Attention Runtime → Attention Operators: selects operator (evidence: `src/micro_vllm/attention/attention.py`)
- Model Runner → Sampler: samples logits (evidence: `src/micro_vllm/worker/model_runner.py`)
- Sampler → Sampling Parameters: applies policy (evidence: `src/micro_vllm/sampling/sampler.py`)
- LLM API → Engine Configuration: configures engine (evidence: `src/micro_vllm/llm.py`)
- Model Runner → Engine Configuration: reads settings (evidence: `src/micro_vllm/worker/model_runner.py`)
- Engine Configuration → Trace Sink: accepts trace sink (evidence: `src/micro_vllm/config.py`)