# huyc0928/mini-sglang architecture

> Architecture diagram of the GitHub repository huyc0928/mini-sglang, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/huyc0928/mini-sglang
- Repository: https://github.com/huyc0928/mini-sglang

## Overview

Mini-SGLang is a lightweight LLM inference and serving framework. Online requests enter through the OpenAI-compatible API, are tokenized, scheduled as prefill/decode work, executed by the model engine, then detokenized and returned. The graph also shows offline generation and major serving optimizations documented in the README. Some scheduler and API wiring is unsampled; those relationships are shown only where supported by the excerpts or README.

Main entry points: [`python/minisgl/scheduler/scheduler.py`](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/scheduler/scheduler.py) (Scheduler), [`python/minisgl/tokenizer/server.py`](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/tokenizer/server.py) (Tokenizer worker), [`python/minisgl/engine/engine.py`](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/engine/engine.py) (Inference engine), [`python/minisgl/server/api_server.py`](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/server/api_server.py) (API server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User interfaces"]
  node_api["API server<br/>[api_server.py]"]
  node_shell["Interactive shell<br/>[shell.py]"]
  node_offline["Offline LLM<br/>[llm.py]"]
end

subgraph group_orchestration["Request orchestration"]
  node_tokenizer["Tokenizer worker<br/>[server.py]"]
  node_messages["Message protocols"]
  node_scheduler["Scheduler<br/>[scheduler.py]"]
  node_prefill["Prefill manager<br/>[prefill.py]"]
  node_decode["Decode manager<br/>[decode.py]"]
  node_table["Page table<br/>[table.py]"]
end

subgraph group_execution["Model execution"]
  node_engine["Inference engine<br/>[engine.py]"]
  node_model["Model implementations"]
  node_layers["Transformer layers"]
  node_attention["Attention backends"]
  node_kernels["Optimized kernels"]
  node_moe["MoE execution"]
end

subgraph group_optimization["Serving optimizations"]
  node_cache[("KV cache pools")]
  node_prefix[("Radix prefix cache<br/>[radix_cache.py]")]
  node_sampling["Sampling and graph"]
end

subgraph group_distributed["Parallel execution"]
  node_distributed["Tensor parallelism"]
end

node_client(("API client"))
node_cuda(("CUDA GPU"))

node_client -->|"sends request"| node_api
node_api -->|"sends prompt"| node_tokenizer
node_api -->|"uses messages"| node_messages
node_tokenizer -->|"encodes tokens"| node_messages
node_tokenizer -->|"dispatches tokens"| node_scheduler
node_scheduler -->|"adds requests"| node_prefill
node_scheduler -->|"manages decode"| node_decode
node_scheduler -->|"updates pages"| node_table
node_scheduler -->|"manages cache"| node_cache
node_scheduler -->|"forwards batches"| node_engine
node_engine -->|"runs forward"| node_model
node_model -->|"composes layers"| node_layers
node_layers -->|"calls attention"| node_attention
node_attention -->|"reads and writes"| node_cache
node_attention -->|"uses kernels"| node_kernels
node_layers -->|"calls MoE"| node_moe
node_moe -->|"uses kernels"| node_kernels
node_prefix -->|"manages prefixes"| node_cache
node_scheduler -->|"caches prefixes"| node_prefix
node_engine -->|"uses graph runner"| node_sampling
node_sampling -->|"executes on"| node_cuda
node_engine -->|"initializes ranks"| node_distributed
node_distributed -->|"coordinates GPUs"| node_cuda
node_tokenizer -->|"returns text"| node_api
node_client -->|"interacts in shell"| node_shell
node_offline -->|"runs requests"| node_scheduler
node_offline -->|"loads tokenizer"| node_model

click node_api "https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/server/api_server.py"
click node_shell "https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/shell.py"
click node_offline "https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/llm/llm.py"
click node_tokenizer "https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/tokenizer/server.py"
click node_messages "https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/message"
click node_scheduler "https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/scheduler/scheduler.py"
click node_prefill "https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/scheduler/prefill.py"
click node_decode "https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/scheduler/decode.py"
click node_table "https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/scheduler/table.py"
click node_engine "https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/engine/engine.py"
click node_model "https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/models"
click node_layers "https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/layers"
click node_attention "https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/attention"
click node_kernels "https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/kernel"
click node_cache "https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/kvcache"
click node_prefix "https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/kvcache/radix_cache.py"
click node_sampling "https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/engine"
click node_moe "https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/moe"
click node_distributed "https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/distributed"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_shell,node_offline,node_client toneBlue
class node_tokenizer,node_messages,node_scheduler,node_prefill,node_decode,node_table toneAmber
class node_engine,node_model,node_layers,node_attention,node_kernels,node_moe toneMint
class node_cache,node_prefix,node_sampling toneRose
class node_distributed,node_cuda toneIndigo
```

## Components

### User interfaces

- [API server](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/server/api_server.py): `python/minisgl/server/api_server.py`
- [Interactive shell](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/shell.py): `python/minisgl/shell.py`
- [Offline LLM](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/llm/llm.py): `python/minisgl/llm/llm.py`

### Request orchestration

- [Tokenizer worker](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/tokenizer/server.py): `python/minisgl/tokenizer/server.py`
- [Message protocols](https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/message): `python/minisgl/message`
- [Scheduler](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/scheduler/scheduler.py): `python/minisgl/scheduler/scheduler.py`
- [Prefill manager](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/scheduler/prefill.py): `python/minisgl/scheduler/prefill.py`
- [Decode manager](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/scheduler/decode.py): `python/minisgl/scheduler/decode.py`
- [Page table](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/scheduler/table.py): `python/minisgl/scheduler/table.py`

### Model execution

- [Inference engine](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/engine/engine.py): `python/minisgl/engine/engine.py`
- [Model implementations](https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/models): `python/minisgl/models`
- [Transformer layers](https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/layers): `python/minisgl/layers`
- [Attention backends](https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/attention): `python/minisgl/attention`
- [Optimized kernels](https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/kernel): `python/minisgl/kernel`
- [MoE execution](https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/moe): `python/minisgl/moe`

### Serving optimizations

- [KV cache pools](https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/kvcache): `python/minisgl/kvcache`
- [Radix prefix cache](https://github.com/huyc0928/mini-sglang/blob/main/python/minisgl/kvcache/radix_cache.py): `python/minisgl/kvcache/radix_cache.py`
- [Sampling and graph](https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/engine): `python/minisgl/engine`

### Parallel execution

- [Tensor parallelism](https://github.com/huyc0928/mini-sglang/tree/main/python/minisgl/distributed): `python/minisgl/distributed`

### Other components

- **API client**
- **CUDA GPU**

## Connections

- API client → API server: sends request
- API server → Tokenizer worker: sends prompt
- API server → Message protocols: uses messages
- Tokenizer worker → Message protocols: encodes tokens
- Tokenizer worker → Scheduler: dispatches tokens
- Scheduler → Prefill manager: adds requests
- Scheduler → Decode manager: manages decode
- Scheduler → Page table: updates pages
- Scheduler → KV cache pools: manages cache
- Scheduler → Inference engine: forwards batches
- Inference engine → Model implementations: runs forward
- Model implementations → Transformer layers: composes layers
- Transformer layers → Attention backends: calls attention
- Attention backends → KV cache pools: reads and writes
- Attention backends → Optimized kernels: uses kernels
- Transformer layers → MoE execution: calls MoE
- MoE execution → Optimized kernels: uses kernels
- Radix prefix cache → KV cache pools: manages prefixes
- Scheduler → Radix prefix cache: caches prefixes
- Inference engine → Sampling and graph: uses graph runner
- Sampling and graph → CUDA GPU: executes on
- Inference engine → Tensor parallelism: initializes ranks
- Tensor parallelism → CUDA GPU: coordinates GPUs
- Tokenizer worker → API server: returns text
- API client → Interactive shell: interacts in shell
- Offline LLM → Scheduler: runs requests
- Offline LLM → Model implementations: loads tokenizer