# brave30/llm-inference-engine architecture

> Architecture diagram of the GitHub repository brave30/llm-inference-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/brave30/llm-inference-engine
- Repository: https://github.com/brave30/llm-inference-engine

## Overview

This repository is a GPU LLM inference runtime. Its primary workflow accepts tokenized prompts, schedules mixed decode and prefill work, allocates paged KV-cache space, runs Llama forward passes and attention kernels, then samples and returns generated tokens. The graph also retains weight/config loading and the major kernel and CUDA-graph paths. Evidence is limited to the README and indexed excerpts; unsampled implementation details are omitted where relationships are not documented.

Main entry points: [`inference_engine/engine.py`](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/engine.py) (Inference engine), [`inference_engine/model.py`](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/model.py) (Llama decoder), [`inference_engine/scheduler.py`](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/scheduler.py) (Batch scheduler), [`inference_engine/block_manager.py`](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/block_manager.py) (KV page allocator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Request API"]
  node_engine["Inference engine<br/>[engine.py]"]
  node_tokenizer["Tokenizer"]
end

subgraph group_orchestration["Batch orchestration"]
  node_sequence["Request sequence<br/>[sequence.py]"]
  node_scheduler["Batch scheduler<br/>[scheduler.py]"]
  node_allocator["KV page allocator<br/>[block_manager.py]"]
end

subgraph group_model["Model execution"]
  node_model["Llama decoder<br/>[model.py]"]
  node_kv[("Paged KV cache<br/>[model.py]")]
  node_rmsnorm{{"Fused RMSNorm<br/>[rmsnorm.py]"}}
  node_decode{{"Paged decode attention"}}
  node_prefill["Prefill SDPA<br/>[model.py]"]
  node_sampling["Token sampling<br/>[sampling.py]"]
end

subgraph group_runtime["Runtime resources"]
  node_config["Engine configuration<br/>[config.py]"]
  node_weights["Model weights<br/>[weights.py]"]
  node_hub["Model repository"]
  node_gpu["CUDA GPU"]
end

node_client(("Caller"))

node_client -->|"submits prompts"| node_engine
node_engine -->|"encodes prompts"| node_tokenizer
node_engine -->|"creates sequences"| node_sequence
node_engine -->|"schedules work"| node_scheduler
node_scheduler -->|"reserves pages"| node_allocator
node_engine -->|"runs forward pass"| node_model
node_model -->|"writes and reads"| node_kv
node_model -->|"normalizes"| node_rmsnorm
node_model -->|"dispatches decode"| node_decode
node_model -->|"runs prefill"| node_prefill
node_engine -->|"samples logits"| node_sampling
node_engine -->|"reads settings"| node_config
node_engine -->|"loads weights"| node_weights
node_weights -->|"downloads model"| node_hub
node_engine -.->|"loads tokenizer"| node_tokenizer
node_engine -->|"executes on"| node_gpu
node_engine -->|"returns outputs"| node_client
node_allocator -->|"updates block table"| node_sequence

click node_engine "https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/engine.py"
click node_sequence "https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/sequence.py"
click node_scheduler "https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/scheduler.py"
click node_allocator "https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/block_manager.py"
click node_config "https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/config.py"
click node_weights "https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/weights.py"
click node_model "https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/model.py"
click node_kv "https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/model.py"
click node_rmsnorm "https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/kernels/rmsnorm.py"
click node_decode "https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/kernels/decode_attention.py"
click node_prefill "https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/model.py"
click node_sampling "https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/sampling.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_engine,node_tokenizer toneBlue
class node_sequence,node_scheduler,node_allocator toneAmber
class node_model,node_kv,node_rmsnorm,node_decode,node_prefill,node_sampling toneMint
class node_config,node_weights,node_hub,node_gpu toneRose
class node_client toneIndigo
```

## Components

### Request API

- [Inference engine](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/engine.py): `inference_engine/engine.py`
- **Tokenizer**

### Batch orchestration

- [Request sequence](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/sequence.py): `inference_engine/sequence.py`
- [Batch scheduler](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/scheduler.py): `inference_engine/scheduler.py`
- [KV page allocator](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/block_manager.py): `inference_engine/block_manager.py`

### Model execution

- [Llama decoder](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/model.py): `inference_engine/model.py`
- [Paged KV cache](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/model.py): `inference_engine/model.py`
- [Fused RMSNorm](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/kernels/rmsnorm.py): `inference_engine/kernels/rmsnorm.py`
- [Paged decode attention](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/kernels/decode_attention.py): `inference_engine/kernels/decode_attention.py`
- [Prefill SDPA](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/model.py): `inference_engine/model.py`
- [Token sampling](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/sampling.py): `inference_engine/sampling.py`

### Runtime resources

- [Engine configuration](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/config.py): `inference_engine/config.py`
- [Model weights](https://github.com/brave30/llm-inference-engine/blob/main/inference_engine/weights.py): `inference_engine/weights.py`
- **Model repository**
- **CUDA GPU**

### Other components

- **Caller**

## Connections

- Caller → Inference engine: submits prompts (evidence: `README.md`)
- Inference engine → Tokenizer: encodes prompts (evidence: `README.md`)
- Inference engine → Request sequence: creates sequences (evidence: `inference_engine/engine.py`)
- Inference engine → Batch scheduler: schedules work (evidence: `inference_engine/engine.py`)
- Batch scheduler → KV page allocator: reserves pages (evidence: `inference_engine/scheduler.py`)
- Inference engine → Llama decoder: runs forward pass (evidence: `inference_engine/engine.py`)
- Llama decoder → Paged KV cache: writes and reads (evidence: `inference_engine/model.py`)
- Llama decoder → Fused RMSNorm: normalizes (evidence: `inference_engine/model.py`)
- Llama decoder → Paged decode attention: dispatches decode (evidence: `inference_engine/model.py`)
- Llama decoder → Prefill SDPA: runs prefill (evidence: `inference_engine/model.py`)
- Inference engine → Token sampling: samples logits (evidence: `inference_engine/engine.py`)
- Inference engine → Engine configuration: reads settings (evidence: `inference_engine/engine.py`)
- Inference engine → Model weights: loads weights (evidence: `inference_engine/engine.py`)
- Model weights → Model repository: downloads model (evidence: `inference_engine/weights.py`)
- Inference engine → Tokenizer: loads tokenizer (evidence: `inference_engine/engine.py`)
- Inference engine → CUDA GPU: executes on (evidence: `README.md`)
- Inference engine → Caller: returns outputs (evidence: `README.md`)
- KV page allocator → Request sequence: updates block table (evidence: `inference_engine/block_manager.py`)
