# brave30/mini-inference-engine architecture

> Architecture diagram of the GitHub repository brave30/mini-inference-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/brave30/mini-inference-engine
- Repository: https://github.com/brave30/mini-inference-engine

## Overview

minfer is a PyTorch and Triton Llama inference engine. Its principal workflow accepts prompts, schedules continuous batches of decode tokens and prefill chunks, runs a flattened model forward over paged KV state, samples tokens, and returns generated outputs. The graph also shows weight/tokenizer acquisition, memory allocation and preemption, optimized attention/normalization, and CUDA-graph replay. The README documents these capabilities; sampled source excerpts are partial, so uncertain internal wiring is omitted.

Main entry points: [`minfer/engine.py`](https://github.com/brave30/mini-inference-engine/blob/main/minfer/engine.py) (LLM engine), [`minfer/model.py`](https://github.com/brave30/mini-inference-engine/blob/main/minfer/model.py) (Llama forward), [`minfer/scheduler.py`](https://github.com/brave30/mini-inference-engine/blob/main/minfer/scheduler.py) (Batch scheduler), [`minfer/weights.py`](https://github.com/brave30/mini-inference-engine/blob/main/minfer/weights.py) (Weight loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Request API"]
  node_engine["LLM engine<br/>[engine.py]"]
  node_sampling["Sampling params<br/>[sampling.py]"]
  node_output["Request output<br/>[engine.py]"]
end

subgraph group_runtime["Batch Runtime"]
  node_scheduler["Batch scheduler<br/>[scheduler.py]"]
  node_sequence["Sequence state<br/>[sequence.py]"]
  node_allocator["Page allocator<br/>[block_manager.py]"]
  node_graphs["CUDA graph replay<br/>[engine.py]"]
end

subgraph group_model["Model Compute"]
  node_model["Llama forward<br/>[model.py]"]
  node_prefill["Prefill attention<br/>[model.py]"]
  node_decode["Paged decode attention"]
  node_norm["Fused RMSNorm<br/>[rmsnorm.py]"]
end

subgraph group_resources["Weights And State"]
  node_cache[("Paged KV cache<br/>[model.py]")]
  node_config["Engine/model config<br/>[config.py]"]
  node_weights["Weight loader<br/>[weights.py]"]
end

node_caller(("Inference caller"))
node_tokenizer["HF tokenizer"]
node_hub["Model hub"]

node_caller -->|"submits prompts"| node_engine
node_caller -->|"sets parameters"| node_sampling
node_engine -.->|"encodes and decodes"| node_tokenizer
node_engine -->|"adds and schedules"| node_scheduler
node_scheduler -->|"tracks work"| node_sequence
node_scheduler -->|"reserves and frees"| node_allocator
node_engine -->|"allocates cache"| node_allocator
node_engine -->|"reads settings"| node_config
node_engine -->|"loads model"| node_weights
node_weights -.->|"downloads snapshots"| node_hub
node_engine -->|"constructs and calls"| node_model
node_engine -->|"captures or replays"| node_graphs
node_graphs -->|"runs decode forward"| node_model
node_model -->|"writes and reads"| node_cache
node_model -->|"dispatches decode"| node_decode
node_model -->|"dispatches prefill"| node_prefill
node_model -->|"applies normalization"| node_norm
node_engine -->|"samples logits"| node_sampling
node_engine -->|"updates tokens"| node_sequence
node_engine -->|"builds results"| node_output
node_engine -->|"returns outputs"| node_caller

click node_engine "https://github.com/brave30/mini-inference-engine/blob/main/minfer/engine.py"
click node_sampling "https://github.com/brave30/mini-inference-engine/blob/main/minfer/sampling.py"
click node_output "https://github.com/brave30/mini-inference-engine/blob/main/minfer/engine.py"
click node_scheduler "https://github.com/brave30/mini-inference-engine/blob/main/minfer/scheduler.py"
click node_sequence "https://github.com/brave30/mini-inference-engine/blob/main/minfer/sequence.py"
click node_allocator "https://github.com/brave30/mini-inference-engine/blob/main/minfer/block_manager.py"
click node_cache "https://github.com/brave30/mini-inference-engine/blob/main/minfer/model.py"
click node_config "https://github.com/brave30/mini-inference-engine/blob/main/minfer/config.py"
click node_weights "https://github.com/brave30/mini-inference-engine/blob/main/minfer/weights.py"
click node_model "https://github.com/brave30/mini-inference-engine/blob/main/minfer/model.py"
click node_prefill "https://github.com/brave30/mini-inference-engine/blob/main/minfer/model.py"
click node_decode "https://github.com/brave30/mini-inference-engine/blob/main/minfer/kernels/decode_attention.py"
click node_norm "https://github.com/brave30/mini-inference-engine/blob/main/minfer/kernels/rmsnorm.py"
click node_graphs "https://github.com/brave30/mini-inference-engine/blob/main/minfer/engine.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_engine,node_sampling,node_output toneBlue
class node_scheduler,node_sequence,node_allocator,node_graphs toneAmber
class node_model,node_prefill,node_decode,node_norm toneMint
class node_cache,node_config,node_weights toneRose
class node_caller,node_tokenizer,node_hub toneIndigo
```

## Components

### Request API

- [LLM engine](https://github.com/brave30/mini-inference-engine/blob/main/minfer/engine.py): `minfer/engine.py`
- [Sampling params](https://github.com/brave30/mini-inference-engine/blob/main/minfer/sampling.py): `minfer/sampling.py`
- [Request output](https://github.com/brave30/mini-inference-engine/blob/main/minfer/engine.py): `minfer/engine.py`

### Batch Runtime

- [Batch scheduler](https://github.com/brave30/mini-inference-engine/blob/main/minfer/scheduler.py): `minfer/scheduler.py`
- [Sequence state](https://github.com/brave30/mini-inference-engine/blob/main/minfer/sequence.py): `minfer/sequence.py`
- [Page allocator](https://github.com/brave30/mini-inference-engine/blob/main/minfer/block_manager.py): `minfer/block_manager.py`
- [CUDA graph replay](https://github.com/brave30/mini-inference-engine/blob/main/minfer/engine.py): `minfer/engine.py`

### Model Compute

- [Llama forward](https://github.com/brave30/mini-inference-engine/blob/main/minfer/model.py): `minfer/model.py`
- [Prefill attention](https://github.com/brave30/mini-inference-engine/blob/main/minfer/model.py): `minfer/model.py`
- [Paged decode attention](https://github.com/brave30/mini-inference-engine/blob/main/minfer/kernels/decode_attention.py): `minfer/kernels/decode_attention.py`
- [Fused RMSNorm](https://github.com/brave30/mini-inference-engine/blob/main/minfer/kernels/rmsnorm.py): `minfer/kernels/rmsnorm.py`

### Weights And State

- [Paged KV cache](https://github.com/brave30/mini-inference-engine/blob/main/minfer/model.py): `minfer/model.py`
- [Engine/model config](https://github.com/brave30/mini-inference-engine/blob/main/minfer/config.py): `minfer/config.py`
- [Weight loader](https://github.com/brave30/mini-inference-engine/blob/main/minfer/weights.py): `minfer/weights.py`

### Other components

- **Inference caller**
- **HF tokenizer**
- **Model hub**

## Connections

- Inference caller → LLM engine: submits prompts
- Inference caller → Sampling params: sets parameters
- LLM engine → HF tokenizer: encodes and decodes
- LLM engine → Batch scheduler: adds and schedules
- Batch scheduler → Sequence state: tracks work
- Batch scheduler → Page allocator: reserves and frees
- LLM engine → Page allocator: allocates cache
- LLM engine → Engine/model config: reads settings
- LLM engine → Weight loader: loads model
- Weight loader → Model hub: downloads snapshots
- LLM engine → Llama forward: constructs and calls
- LLM engine → CUDA graph replay: captures or replays
- CUDA graph replay → Llama forward: runs decode forward
- Llama forward → Paged KV cache: writes and reads
- Llama forward → Paged decode attention: dispatches decode
- Llama forward → Prefill attention: dispatches prefill
- Llama forward → Fused RMSNorm: applies normalization
- LLM engine → Sampling params: samples logits
- LLM engine → Sequence state: updates tokens
- LLM engine → Request output: builds results
- LLM engine → Inference caller: returns outputs