# w2315516642/mini-vllm architecture

> Architecture diagram of the GitHub repository w2315516642/mini-vllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/w2315516642/mini-vllm
- Repository: https://github.com/w2315516642/mini-vllm

## Overview

MiniVLLM is a Python LLM inference runtime: callers submit prompts and sampling settings, then receive generated text and token-level results. The engine schedules prompt and decode work, tracks sequence state and KV blocks, dispatches workers for model execution, and collects outputs. The sampled code supports prefix-aware KV caching and tensor-parallel LLaMA inference; the tree also shows Ray utilities for distributed execution. The README is empty, and unsampled subsystems are included only where their paths and roles are evident from the tree.

Main entry points: [`minivllm/engine/llm_engine.py`](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/engine/llm_engine.py) (LLM Engine), [`minivllm/entrypoints/llm.py`](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/entrypoints/llm.py) (LLM API), [`minivllm/kv_cache/scheduler.py`](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/kv_cache/scheduler.py) (Scheduler), [`minivllm/engine/model_runner.py`](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/engine/model_runner.py) (Model Runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Public API"]
  node_llm["LLM API<br/>[llm.py]"]
  node_tokenizer["Tokenizer<br/>[tokenizer_utils.py]"]
  node_sampling["Sampling Params<br/>[sampling_params.py]"]
  node_outputs["Request Outputs<br/>[outputs.py]"]
end

subgraph group_runtime["Engine Runtime"]
  node_engine["LLM Engine<br/>[llm_engine.py]"]
  node_scheduler["Scheduler<br/>[scheduler.py]"]
  node_worker["Worker<br/>[worker.py]"]
  node_ray["Ray Runtime<br/>[ray_utils.py]"]
end

subgraph group_cache["Sequence &amp; Cache"]
  node_sequences["Sequences<br/>[sequence.py]"]
  node_blocks["Block Manager<br/>[block_manager.py]"]
  node_policy["Scheduling Policy<br/>[policy.py]"]
  node_cacheengine[("KV Cache Engine<br/>[cache_engine.py]")]
end

subgraph group_inference["Model Inference"]
  node_runner["Model Runner<br/>[model_runner.py]"]
  node_metadata["Input Metadata<br/>[input_metadata.py]"]
  node_loader["Model Loader<br/>[model_loader.py]"]
  node_llama["LLaMA Model<br/>[llama.py]"]
  node_sampler["Token Sampler<br/>[sampler.py]"]
  node_attention["Attention Layer<br/>[attention.py]"]
  node_kernels["Attention Kernels<br/>[attention.cpp]"]
end

node_caller(("Application"))

node_caller -->|"submits prompts"| node_llm
node_llm -->|"adds requests"| node_engine
node_llm -->|"steps engine"| node_engine
node_llm -->|"encodes prompts"| node_tokenizer
node_llm -->|"decodes results"| node_tokenizer
node_llm -->|"sets sampling"| node_sampling
node_engine -->|"schedules work"| node_scheduler
node_engine -->|"creates sequences"| node_sequences
node_engine -->|"dispatches execution"| node_worker
node_engine -->|"builds results"| node_outputs
node_scheduler -->|"tracks sequence state"| node_sequences
node_scheduler -->|"allocates blocks"| node_blocks
node_scheduler -->|"selects work"| node_policy
node_runner -->|"builds inputs"| node_metadata
node_worker -->|"runs model"| node_runner
node_worker -->|"manages cache"| node_cacheengine
node_runner -->|"loads model"| node_loader
node_runner -->|"executes forward"| node_llama
node_llama -->|"computes attention"| node_attention
node_llama -->|"samples tokens"| node_sampler
node_attention -->|"invokes kernels"| node_kernels
node_engine -.->|"uses distributed workers"| node_ray

click node_llm "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/entrypoints/llm.py"
click node_engine "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/engine/llm_engine.py"
click node_tokenizer "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/engine/tokenizer_utils.py"
click node_sampling "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/sampling_params.py"
click node_outputs "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/outputs.py"
click node_sequences "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/sequence.py"
click node_scheduler "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/kv_cache/scheduler.py"
click node_blocks "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/kv_cache/block_manager.py"
click node_policy "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/kv_cache/policy.py"
click node_runner "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/engine/model_runner.py"
click node_worker "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/worker/worker.py"
click node_cacheengine "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/worker/cache_engine.py"
click node_metadata "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/model_executor/input_metadata.py"
click node_loader "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/model_executor/model_loader.py"
click node_llama "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/model_executor/models/llama.py"
click node_sampler "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/model_executor/layers/sampler.py"
click node_attention "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/model_executor/layers/attention.py"
click node_kernels "https://github.com/w2315516642/mini-vllm/blob/master/csrc/attention.cpp"
click node_ray "https://github.com/w2315516642/mini-vllm/blob/master/minivllm/engine/ray_utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_llm,node_tokenizer,node_sampling,node_outputs toneBlue
class node_engine,node_scheduler,node_worker,node_ray toneAmber
class node_sequences,node_blocks,node_policy,node_cacheengine toneMint
class node_runner,node_metadata,node_loader,node_llama,node_sampler,node_attention,node_kernels toneRose
class node_caller toneIndigo
```

## Components

### Public API

- [LLM API](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/entrypoints/llm.py): `minivllm/entrypoints/llm.py`
- [Tokenizer](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/engine/tokenizer_utils.py): `minivllm/engine/tokenizer_utils.py`
- [Sampling Params](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/sampling_params.py): `minivllm/sampling_params.py`
- [Request Outputs](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/outputs.py): `minivllm/outputs.py`

### Engine Runtime

- [LLM Engine](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/engine/llm_engine.py): `minivllm/engine/llm_engine.py`
- [Scheduler](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/kv_cache/scheduler.py): `minivllm/kv_cache/scheduler.py`
- [Worker](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/worker/worker.py): `minivllm/worker/worker.py`
- [Ray Runtime](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/engine/ray_utils.py): `minivllm/engine/ray_utils.py`

### Sequence & Cache

- [Sequences](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/sequence.py): `minivllm/sequence.py`
- [Block Manager](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/kv_cache/block_manager.py): `minivllm/kv_cache/block_manager.py`
- [Scheduling Policy](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/kv_cache/policy.py): `minivllm/kv_cache/policy.py`
- [KV Cache Engine](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/worker/cache_engine.py): `minivllm/worker/cache_engine.py`

### Model Inference

- [Model Runner](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/engine/model_runner.py): `minivllm/engine/model_runner.py`
- [Input Metadata](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/model_executor/input_metadata.py): `minivllm/model_executor/input_metadata.py`
- [Model Loader](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/model_executor/model_loader.py): `minivllm/model_executor/model_loader.py`
- [LLaMA Model](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/model_executor/models/llama.py): `minivllm/model_executor/models/llama.py`
- [Token Sampler](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/model_executor/layers/sampler.py): `minivllm/model_executor/layers/sampler.py`
- [Attention Layer](https://github.com/w2315516642/mini-vllm/blob/master/minivllm/model_executor/layers/attention.py): `minivllm/model_executor/layers/attention.py`
- [Attention Kernels](https://github.com/w2315516642/mini-vllm/blob/master/csrc/attention.cpp): `csrc/attention.cpp`

### Other components

- **Application**

## Connections

- Application → LLM API: submits prompts
- LLM API → LLM Engine: adds requests
- LLM API → LLM Engine: steps engine
- LLM API → Tokenizer: encodes prompts
- LLM API → Tokenizer: decodes results
- LLM API → Sampling Params: sets sampling
- LLM Engine → Scheduler: schedules work
- LLM Engine → Sequences: creates sequences
- LLM Engine → Worker: dispatches execution
- LLM Engine → Request Outputs: builds results
- Scheduler → Sequences: tracks sequence state
- Scheduler → Block Manager: allocates blocks
- Scheduler → Scheduling Policy: selects work
- Model Runner → Input Metadata: builds inputs
- Worker → Model Runner: runs model
- Worker → KV Cache Engine: manages cache
- Model Runner → Model Loader: loads model
- Model Runner → LLaMA Model: executes forward
- LLaMA Model → Attention Layer: computes attention
- LLaMA Model → Token Sampler: samples tokens
- Attention Layer → Attention Kernels: invokes kernels
- LLM Engine → Ray Runtime: uses distributed workers