nopnoping/micro-vllm

How nopnoping/micro-vllm works

micro-vLLM is a lightweight, synchronous inference engine for decoder-only language models. The principal workflow accepts prompts through the public LLM API, tokenizes them, schedules requests under token and sequence budgets, manages logical KV-cache pages, executes immutable plans through an executor and worker, then samples and decodes generated tokens. The map also preserves model loading, supported architectures, replaceable attention backends, and tracing. Relationships are limited to README statements and shown source references/calls; some internal implementations were not excerpted, so uncertain links are omitted.

Main entry points: llm.py LLM API · engine.py Engine · attention.py Attention Runtime · scheduler.py Scheduler

Components 28

Public API

LLM API src/micro_vllm/llm.py
Tokenizer src/micro_vllm/tokenizer.py
Sampling Parameters src/micro_vllm/sampling/params.py
Generation Output src/micro_vllm/engine/output.py

Request Control

Engine src/micro_vllm/engine/engine.py
Scheduler src/micro_vllm/core/scheduler.py
Request State src/micro_vllm/core/request.py
Execution Plan src/micro_vllm/core/scheduler_output.py
KV Cache Manager src/micro_vllm/core/kv_cache_manager.py
Cache Groups src/micro_vllm/core/cache_group_manager.py
Cache Blocks src/micro_vllm/core/block.py
Prefix Cache src/micro_vllm/core/prefix_cache.py

Model Runtime

Executor src/micro_vllm/executor/single_process.py
Worker src/micro_vllm/worker/worker.py
Model Runner src/micro_vllm/worker/model_runner.py
Model Loader src/micro_vllm/model_loader/default.py
Sampler src/micro_vllm/sampling/sampler.py

Models and Operators

Model Registry src/micro_vllm/models/registry.py
Llama src/micro_vllm/models/llama.py
Mistral src/micro_vllm/models/mistral.py
Qwen3 src/micro_vllm/models/qwen.py
Model Layers src/micro_vllm/layers
Attention Runtime src/micro_vllm/attention/attention.py
Attention Semantics src/micro_vllm/attention/spec.py
Attention Operators src/micro_vllm/attention/operators/registry.py

Configuration and Trace

Engine Configuration src/micro_vllm/config.py
Trace Sink src/micro_vllm/trace.py

Other components

Caller

Connections 34

Caller → LLM API: submits prompts
LLM API → Tokenizer: tokenizes prompts src/micro_vllm/llm.py
LLM API → Engine: runs generation src/micro_vllm/llm.py
LLM API → Sampling Parameters: sets policy src/micro_vllm/llm.py
Engine → Scheduler: schedules requests src/micro_vllm/engine/engine.py
Engine → Request State: creates requests src/micro_vllm/engine/engine.py
Engine → Executor: executes plan src/micro_vllm/engine/engine.py
Engine → Generation Output: builds outputs src/micro_vllm/engine/engine.py
Scheduler → KV Cache Manager: manages cache src/micro_vllm/core/scheduler.py
Scheduler → Request State: updates state src/micro_vllm/core/scheduler.py
Scheduler → Execution Plan: produces plan src/micro_vllm/core/scheduler.py
KV Cache Manager → Cache Groups: coordinates groups src/micro_vllm/core/kv_cache_manager.py
Cache Groups → Cache Blocks: allocates blocks src/micro_vllm/core/cache_group_manager.py
Cache Groups → Prefix Cache: uses prefix cache src/micro_vllm/core/cache_group_manager.py
Executor → Worker: dispatches plan src/micro_vllm/executor/single_process.py
Worker → Model Runner: delegates execution src/micro_vllm/worker/worker.py
Model Runner → Model Loader: loads model src/micro_vllm/worker/model_runner.py
Model Loader → Model Registry: selects architecture
Model Registry → Llama: resolves model src/micro_vllm/models/registry.py
Model Registry → Mistral: resolves model src/micro_vllm/models/registry.py
Model Registry → Qwen3: resolves model src/micro_vllm/models/registry.py
Llama → Model Layers: uses layers src/micro_vllm/models/llama.py
Mistral → Model Layers: uses layers src/micro_vllm/models/mistral.py
Qwen3 → Model Layers: uses layers src/micro_vllm/models/qwen.py
Llama → Attention Runtime: runs attention src/micro_vllm/models/llama.py
Mistral → Attention Runtime: runs attention src/micro_vllm/models/mistral.py
Qwen3 → Attention Runtime: runs attention src/micro_vllm/models/qwen.py
Attention Runtime → Attention Semantics: builds visibility src/micro_vllm/attention/attention.py
Attention Runtime → Attention Operators: selects operator src/micro_vllm/attention/attention.py
Model Runner → Sampler: samples logits src/micro_vllm/worker/model_runner.py
Sampler → Sampling Parameters: applies policy src/micro_vllm/sampling/sampler.py
LLM API → Engine Configuration: configures engine src/micro_vllm/llm.py
Model Runner → Engine Configuration: reads settings src/micro_vllm/worker/model_runner.py
Engine Configuration → Trace Sink: accepts trace sink src/micro_vllm/config.py

More Python diagrams: nahueespinosa/ai50 · niklastiede/github-trending-api · njprovk/creditscardtools · one-f-m/one_fm · paillat-dev/viralfactory

Drawn by GitDiagram on November 9, 2026 from the repository's file tree, README and source files. Read as Markdown ·