How jvantuyl/strix-halo-sglang works
This repository packages SGLang as an AMD gfx1151 LLM server. A client submits chat or tool requests to the launcher's HTTP service; the scheduler batches work, applies prefix caching, loads a Hugging Face model, executes attention/MoE/quantized kernels, and returns generated tokens. The repository mainly supplies ROCm compatibility patches, Qwen4-Exp PLE handling, conversion utilities, and tuned kernel configurations; the underlying SGLang runtime is upstream and therefore unsampled here.
Components 26
Serving
- Server Launcher
start-sglang.sh - HTTP Server
- Request Scheduler
patches/patch_sleep_on_idle.py - Continuous Batching
- Prefix Cache
- Tool Calling
Model Execution
- Inference Engine
- Model Loader
- Attention Kernels
patches/patch_qwen4_exp_rocm.py - MoE Kernels
patches/patch_wna16_zp.py - Vision Inputs
patches/patch_qwen4_exp_rocm.py
ROCm Backends
- Dense Int4
patches/patch_wna16_rocm.py - AWQ MoE
patches/patch_wna16_zp.py - MXFP4 Backend
patches/fix_aiter_gfx1151_mxfp4.py - Native LayerNorm
patches/03-awq-moe-triton-dispatch.md
PLE Features
- PLE Runtime
patches/patch_qwen4_exp_rocm.py - PLE FP8 Conversion
tools/convert_ple_fp8.py - PLE Table Reuse
patches/patch_ple_table_reuse.py - CUDA Graph Replay
patches/patch_cuda_graph_ple.py
Optimization Assets
- Tuned MoE Configs
configs/moe - Weight Quantizer
tools/quantize_nonexpert.py - Model Cache
README.md
Other components
- API Client
- Hugging Face Models
- gfx1151 GPU
- External Tools
Connections 27
- API Client → HTTP Server: sends requests
- Server Launcher → HTTP Server: starts server
- HTTP Server → Request Scheduler: dispatches requests
- HTTP Server → Tool Calling: parses tools
- Tool Calling → External Tools: calls tools
- Request Scheduler → Continuous Batching: forms batches
- Request Scheduler → Prefix Cache: checks prefixes
- Request Scheduler → Inference Engine: runs batches
- Inference Engine → Model Loader: loads model
- Model Loader → Hugging Face Models: downloads weights
- Model Loader → Model Cache: uses cache
- Inference Engine → Attention Kernels: runs attention
- Inference Engine → MoE Kernels: runs MoE
- Inference Engine → Vision Inputs: processes vision
- Inference Engine → Dense Int4: runs dense layers
- Inference Engine → AWQ MoE: runs AWQ experts
- Inference Engine → MXFP4 Backend: runs MXFP4
- Inference Engine → Native LayerNorm: normalizes states
- Inference Engine → PLE Runtime: reads PLE
- PLE FP8 Conversion → PLE Table Reuse: creates tables
- Model Loader → PLE Table Reuse: reuses tables
- PLE Runtime → CUDA Graph Replay: supports replay
- Tuned MoE Configs → MoE Kernels: configures tiles
- Weight Quantizer → Hugging Face Models: rewrites weights
- Inference Engine → gfx1151 GPU: executes kernels
- Inference Engine → HTTP Server: returns tokens
- HTTP Server → API Client: returns results