How huyc0928/mini-sglang works
Mini-SGLang is a lightweight LLM inference and serving framework. Online requests enter through the OpenAI-compatible API, are tokenized, scheduled as prefill/decode work, executed by the model engine, then detokenized and returned. The graph also shows offline generation and major serving optimizations documented in the README. Some scheduler and API wiring is unsampled; those relationships are shown only where supported by the excerpts or README.
Main entry points: scheduler.py Scheduler · server.py Tokenizer worker · engine.py Inference engine · api_server.py API server
Components 21
User interfaces
- API server
python/minisgl/server/api_server.py - Interactive shell
python/minisgl/shell.py - Offline LLM
python/minisgl/llm/llm.py
Request orchestration
- Tokenizer worker
python/minisgl/tokenizer/server.py - Message protocols
python/minisgl/message - Scheduler
python/minisgl/scheduler/scheduler.py - Prefill manager
python/minisgl/scheduler/prefill.py - Decode manager
python/minisgl/scheduler/decode.py - Page table
python/minisgl/scheduler/table.py
Model execution
- Inference engine
python/minisgl/engine/engine.py - Model implementations
python/minisgl/models - Transformer layers
python/minisgl/layers - Attention backends
python/minisgl/attention - Optimized kernels
python/minisgl/kernel - MoE execution
python/minisgl/moe
Serving optimizations
- KV cache pools
python/minisgl/kvcache - Radix prefix cache
python/minisgl/kvcache/radix_cache.py - Sampling and graph
python/minisgl/engine
Parallel execution
- Tensor parallelism
python/minisgl/distributed
Other components
- API client
- CUDA GPU
Connections 27
- API client → API server: sends request
- API server → Tokenizer worker: sends prompt
- API server → Message protocols: uses messages
- Tokenizer worker → Message protocols: encodes tokens
- Tokenizer worker → Scheduler: dispatches tokens
- Scheduler → Prefill manager: adds requests
- Scheduler → Decode manager: manages decode
- Scheduler → Page table: updates pages
- Scheduler → KV cache pools: manages cache
- Scheduler → Inference engine: forwards batches
- Inference engine → Model implementations: runs forward
- Model implementations → Transformer layers: composes layers
- Transformer layers → Attention backends: calls attention
- Attention backends → KV cache pools: reads and writes
- Attention backends → Optimized kernels: uses kernels
- Transformer layers → MoE execution: calls MoE
- MoE execution → Optimized kernels: uses kernels
- Radix prefix cache → KV cache pools: manages prefixes
- Scheduler → Radix prefix cache: caches prefixes
- Inference engine → Sampling and graph: uses graph runner
- Sampling and graph → CUDA GPU: executes on
- Inference engine → Tensor parallelism: initializes ranks
- Tensor parallelism → CUDA GPU: coordinates GPUs
- Tokenizer worker → API server: returns text
- API client → Interactive shell: interacts in shell
- Offline LLM → Scheduler: runs requests
- Offline LLM → Model implementations: loads tokenizer