# goabiaryan/smol_vllm architecture

> Architecture diagram of the GitHub repository goabiaryan/smol_vllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/goabiaryan/smol_vllm
- Repository: https://github.com/goabiaryan/smol_vllm

## Overview

smol-vllm is an educational paged-attention inference engine, usable with a deterministic fake backend or optional Hugging Face causal models. A caller submits token IDs to LLMEngine; the engine creates sequence state, schedules batches, allocates KV block tables, runs prefill/decode, appends generated tokens, and emits per-step outputs while collecting metrics. Scheduling and cache allocation support batching, preemption/swap bookkeeping, and block sharing. Metrics and the experiment demo expose latency, throughput, memory pressure, and prefix-sharing behavior. Edges reflect shown calls or README capabilities; the actual swap integration with model KV state is not shown.

Main entry points: [`smol_vllm/engine.py`](https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/engine.py) (LLM engine), [`smol_vllm/demo.py`](https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/demo.py) (Experiments demo), [`smol_vllm/block_manager.py`](https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/block_manager.py) (KV block manager), [`smol_vllm/scheduler.py`](https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/scheduler.py) (Batch scheduler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Request and orchestration"]
  node_engine["LLM engine<br/>[engine.py]"]
  node_sequence["Sequence state<br/>[sequence.py]"]
end

subgraph group_runtime["Scheduling and KV memory"]
  node_scheduler["Batch scheduler<br/>[scheduler.py]"]
  node_blocks[("KV block manager<br/>[block_manager.py]")]
end

subgraph group_models["Inference backends"]
  node_fake["Fake model<br/>[model.py]"]
  node_causal["CausalLM backend<br/>[causal_model.py]"]
end

subgraph group_observability["Metrics and demos"]
  node_metrics["Runtime metrics<br/>[metrics.py]"]
  node_demo["Experiments demo<br/>[demo.py]"]
end

node_caller(("Application caller"))
node_hf{{"Hugging Face models"}}

node_caller -->|"submits prompts"| node_engine
node_engine -->|"creates state"| node_sequence
node_engine -->|"schedules batches"| node_scheduler
node_scheduler -->|"allocates and frees"| node_blocks
node_engine -->|"reads tables; appends tokens"| node_blocks
node_engine -->|"runs prefill/decode"| node_fake
node_engine -->|"optionally selects"| node_causal
node_causal -->|"loads model/tokenizer"| node_hf
node_engine -->|"records timings"| node_metrics
node_demo -->|"runs experiments"| node_engine
node_demo -->|"demonstrates sharing"| node_blocks
node_demo -->|"prints and exports"| node_metrics

click node_engine "https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/engine.py"
click node_sequence "https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/sequence.py"
click node_scheduler "https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/scheduler.py"
click node_blocks "https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/block_manager.py"
click node_fake "https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/model.py"
click node_causal "https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/causal_model.py"
click node_metrics "https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/metrics.py"
click node_demo "https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/demo.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_engine,node_sequence toneBlue
class node_scheduler,node_blocks toneAmber
class node_fake,node_causal toneMint
class node_metrics,node_demo toneRose
class node_caller,node_hf toneIndigo
```

## Components

### Request and orchestration

- [LLM engine](https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/engine.py): `smol_vllm/engine.py`
- [Sequence state](https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/sequence.py): `smol_vllm/sequence.py`

### Scheduling and KV memory

- [Batch scheduler](https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/scheduler.py): `smol_vllm/scheduler.py`
- [KV block manager](https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/block_manager.py): `smol_vllm/block_manager.py`

### Inference backends

- [Fake model](https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/model.py): `smol_vllm/model.py`
- [CausalLM backend](https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/causal_model.py): `smol_vllm/causal_model.py`

### Metrics and demos

- [Runtime metrics](https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/metrics.py): `smol_vllm/metrics.py`
- [Experiments demo](https://github.com/goabiaryan/smol_vllm/blob/main/smol_vllm/demo.py): `smol_vllm/demo.py`

### Other components

- **Application caller**
- **Hugging Face models**

## Connections

- Application caller → LLM engine: submits prompts (evidence: `README.md`)
- LLM engine → Sequence state: creates state (evidence: `smol_vllm/engine.py`)
- LLM engine → Batch scheduler: schedules batches (evidence: `smol_vllm/engine.py`)
- Batch scheduler → KV block manager: allocates and frees (evidence: `smol_vllm/scheduler.py`)
- LLM engine → KV block manager: reads tables; appends tokens (evidence: `smol_vllm/engine.py`)
- LLM engine → Fake model: runs prefill/decode (evidence: `smol_vllm/engine.py`)
- LLM engine → CausalLM backend: optionally selects (evidence: `smol_vllm/engine.py`)
- CausalLM backend → Hugging Face models: loads model/tokenizer (evidence: `smol_vllm/causal_model.py`)
- LLM engine → Runtime metrics: records timings (evidence: `smol_vllm/engine.py`)
- Experiments demo → LLM engine: runs experiments (evidence: `smol_vllm/demo.py`)
- Experiments demo → KV block manager: demonstrates sharing (evidence: `smol_vllm/demo.py`)
- Experiments demo → Runtime metrics: prints and exports (evidence: `smol_vllm/demo.py`)
