# vizuaraai/capstone-mini-inference-engine architecture

> Architecture diagram of the GitHub repository vizuaraai/capstone-mini-inference-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vizuaraai/capstone-mini-inference-engine
- Repository: https://github.com/vizuaraai/capstone-mini-inference-engine

## Overview

NanoServe is a small language-model inference server and a benchmark harness. The main request path accepts text through an OpenAI-compatible API, tokenizes it, schedules continuous batches, runs model inference over a paged KV cache, samples tokens, and streams decoded output. A separate documented benchmark path compares NanoServe with vLLM through the API and reports measured results. The graph follows README and source-index evidence; some internal edges are omitted where excerpts do not establish the caller relationship.

Main entry points: [`nanoserve/engine/llm_engine.py`](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/engine/llm_engine.py) (Engine loop), [`nanoserve/scheduler/scheduler.py`](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/scheduler/scheduler.py) (Continuous scheduler), [`nanoserve/model`](https://github.com/vizuaraai/capstone-mini-inference-engine/tree/main/nanoserve/model) (Model forward passes), [`nanoserve/server/api.py`](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/server/api.py) (OpenAI API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving interface"]
  node_api["OpenAI API<br/>[api.py]"]
  node_protocol["Request protocol<br/>[protocol.py]"]
  node_tokenizer["Tokenizer<br/>[tokenizer.py]"]
end

subgraph group_execution["Inference execution"]
  node_engine["Engine loop<br/>[llm_engine.py]"]
  node_sequence["Sequence state<br/>[request.py]"]
  node_scheduler["Continuous scheduler<br/>[scheduler.py]"]
  node_policy["Scheduling policy<br/>[policy.py]"]
  node_runner["Batch model runner<br/>[model_runner.py]"]
  node_sampler["Token sampler<br/>[sampler.py]"]
  node_model["Model forward passes"]
  node_model_loader["Checkpoint loader<br/>[loader.py]"]
  node_layers["Model layers<br/>[layers.py]"]
end

subgraph group_memory["Paged memory"]
  node_attention["Paged attention<br/>[paged_attention.py]"]
  node_triton["Triton attention<br/>[triton_paged.py]"]
  node_cache[("KV tensor pool<br/>[cache.py]")]
  node_table["Block tables<br/>[block_table.py]"]
  node_allocator["Block allocator<br/>[block_allocator.py]"]
end

subgraph group_measurement["Benchmarking"]
  node_harness["Benchmark harness<br/>[harness.py]"]
  node_metrics["Benchmark metrics<br/>[metrics.py]"]
  node_report["Results report<br/>[report.py]"]
end

node_client(("API client"))
node_vllm["vLLM"]

node_client -->|"sends requests"| node_api
node_api -->|"uses protocol"| node_protocol
node_api -->|"submits generation"| node_engine
node_engine -->|"encodes and decodes"| node_tokenizer
node_engine -->|"loads model"| node_model_loader
node_engine -->|"schedules steps"| node_scheduler
node_engine -->|"runs batches"| node_runner
node_engine -->|"samples tokens"| node_sampler
node_engine -->|"advances sequences"| node_sequence
node_scheduler -->|"schedules"| node_sequence
node_scheduler -->|"applies policy"| node_policy
node_scheduler -->|"allocates blocks"| node_allocator
node_scheduler -->|"grows tables"| node_table
node_table -->|"allocates and frees"| node_allocator
node_runner -->|"runs forward pass"| node_model
node_runner -->|"builds cache metadata"| node_cache
node_model -->|"uses loaded weights"| node_model_loader
node_model -->|"uses operations"| node_layers
node_model -->|"computes attention"| node_attention
node_attention -->|"gathers KV slots"| node_cache
node_triton -->|"gathers KV slots"| node_cache
node_harness -->|"sends benchmark traffic"| node_api
node_harness -->|"benchmarks against"| node_vllm
node_harness -.->|"measures runs"| node_metrics
node_report -.->|"reports measurements"| node_metrics

click node_api "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/server/api.py"
click node_protocol "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/server/protocol.py"
click node_tokenizer "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/tokenizer.py"
click node_engine "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/engine/llm_engine.py"
click node_sequence "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/scheduler/request.py"
click node_scheduler "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/scheduler/scheduler.py"
click node_policy "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/scheduler/policy.py"
click node_runner "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/engine/model_runner.py"
click node_sampler "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/engine/sampler.py"
click node_model "https://github.com/vizuaraai/capstone-mini-inference-engine/tree/main/nanoserve/model"
click node_model_loader "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/model/loader.py"
click node_layers "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/model/layers.py"
click node_attention "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/attention/paged_attention.py"
click node_triton "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/attention/triton_paged.py"
click node_cache "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/kv/cache.py"
click node_table "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/kv/block_table.py"
click node_allocator "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/kv/block_allocator.py"
click node_harness "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/bench/harness.py"
click node_metrics "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/bench/metrics.py"
click node_report "https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/bench/report.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_protocol,node_tokenizer,node_client toneBlue
class node_engine,node_sequence,node_scheduler,node_policy,node_runner,node_sampler,node_model,node_model_loader,node_layers toneAmber
class node_attention,node_triton,node_cache,node_table,node_allocator toneMint
class node_harness,node_metrics,node_report toneRose
class node_vllm toneIndigo
```

## Components

### Serving interface

- [OpenAI API](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/server/api.py): `nanoserve/server/api.py`
- [Request protocol](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/server/protocol.py): `nanoserve/server/protocol.py`
- [Tokenizer](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/tokenizer.py): `nanoserve/tokenizer.py`

### Inference execution

- [Engine loop](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/engine/llm_engine.py): `nanoserve/engine/llm_engine.py`
- [Sequence state](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/scheduler/request.py): `nanoserve/scheduler/request.py`
- [Continuous scheduler](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/scheduler/scheduler.py): `nanoserve/scheduler/scheduler.py`
- [Scheduling policy](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/scheduler/policy.py): `nanoserve/scheduler/policy.py`
- [Batch model runner](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/engine/model_runner.py): `nanoserve/engine/model_runner.py`
- [Token sampler](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/engine/sampler.py): `nanoserve/engine/sampler.py`
- [Model forward passes](https://github.com/vizuaraai/capstone-mini-inference-engine/tree/main/nanoserve/model): `nanoserve/model`
- [Checkpoint loader](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/model/loader.py): `nanoserve/model/loader.py`
- [Model layers](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/model/layers.py): `nanoserve/model/layers.py`

### Paged memory

- [Paged attention](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/attention/paged_attention.py): `nanoserve/attention/paged_attention.py`
- [Triton attention](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/attention/triton_paged.py): `nanoserve/attention/triton_paged.py`
- [KV tensor pool](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/kv/cache.py): `nanoserve/kv/cache.py`
- [Block tables](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/kv/block_table.py): `nanoserve/kv/block_table.py`
- [Block allocator](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/nanoserve/kv/block_allocator.py): `nanoserve/kv/block_allocator.py`

### Benchmarking

- [Benchmark harness](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/bench/harness.py): `bench/harness.py`
- [Benchmark metrics](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/bench/metrics.py): `bench/metrics.py`
- [Results report](https://github.com/vizuaraai/capstone-mini-inference-engine/blob/main/bench/report.py): `bench/report.py`

### Other components

- **API client**
- **vLLM**

## Connections

- API client → OpenAI API: sends requests (evidence: `README.md`)
- OpenAI API → Request protocol: uses protocol (evidence: `nanoserve/server/api.py`)
- OpenAI API → Engine loop: submits generation (evidence: `nanoserve/server/api.py`)
- Engine loop → Tokenizer: encodes and decodes (evidence: `nanoserve/engine/llm_engine.py`)
- Engine loop → Checkpoint loader: loads model (evidence: `nanoserve/engine/llm_engine.py`)
- Engine loop → Continuous scheduler: schedules steps (evidence: `nanoserve/engine/llm_engine.py`)
- Engine loop → Batch model runner: runs batches (evidence: `nanoserve/engine/llm_engine.py`)
- Engine loop → Token sampler: samples tokens (evidence: `nanoserve/engine/llm_engine.py`)
- Engine loop → Sequence state: advances sequences (evidence: `nanoserve/engine/llm_engine.py`)
- Continuous scheduler → Sequence state: schedules (evidence: `nanoserve/scheduler/scheduler.py`)
- Continuous scheduler → Scheduling policy: applies policy (evidence: `nanoserve/scheduler/scheduler.py`)
- Continuous scheduler → Block allocator: allocates blocks (evidence: `nanoserve/scheduler/scheduler.py`)
- Continuous scheduler → Block tables: grows tables (evidence: `nanoserve/scheduler/scheduler.py`)
- Block tables → Block allocator: allocates and frees (evidence: `nanoserve/kv/block_table.py`)
- Batch model runner → Model forward passes: runs forward pass (evidence: `nanoserve/engine/model_runner.py`)
- Batch model runner → KV tensor pool: builds cache metadata (evidence: `nanoserve/engine/model_runner.py`)
- Model forward passes → Checkpoint loader: uses loaded weights (evidence: `nanoserve/model/gemma3.py`)
- Model forward passes → Model layers: uses operations (evidence: `nanoserve/model/gemma3.py`)
- Model forward passes → Paged attention: computes attention (evidence: `nanoserve/model/gemma3.py`)
- Paged attention → KV tensor pool: gathers KV slots (evidence: `nanoserve/attention/paged_attention.py`)
- Triton attention → KV tensor pool: gathers KV slots (evidence: `nanoserve/attention/triton_paged.py`)
- Benchmark harness → OpenAI API: sends benchmark traffic (evidence: `README.md`)
- Benchmark harness → vLLM: benchmarks against (evidence: `README.md`)
- Benchmark harness → Benchmark metrics: measures runs
- Results report → Benchmark metrics: reports measurements
