How slybase/vllm-sly-radiance works
This repository is a gfx1201-focused vLLM fork’s runtime optimization layer, intended to serve quantized Qwen models with fast attention, GEMM and speculative decoding. The principal workflow is a user request entering vLLM’s serving API, running target-model inference with optional draft-and-verify decoding and specialized kernels, then returning generated tokens. Quantization support extends beyond the primary Quark MXFP4 checkpoint to NVFP4, INT4, ParoQuant, AutoRound and escha. The README documents these capabilities; unsampled modules are represented at their exact tree paths, without inferring unshown call relationships.
Main entry points: speculator.py DFlash2 drafter · radiance_paroquant_mxfp4.py ParoQuant MXFP4 · radiance_quant_plugins.py Quant registration · radiance_mxfp4.py MXFP4 dispatch
Components 18
Serving and decoding
- vLLM serving
- DFlash2 drafter
dflash2/speculator.py - Draft head
radiance_drafthead.py - Prompt lookup
sly/radiance_lookup_draft.py - Draft verification
dflash2/qwen3_dflash2.py
Quantization support
- MXFP4 dispatch
radiance_mxfp4.py - NVFP4 support
radiance_nvfp4.py - ParoQuant int4
paroquant/radiance_paroquant.py - ParoQuant MXFP4
paroquant/radiance_paroquant_mxfp4.py - Quant registration
sly/quant/radiance_quant_plugins.py
GPU compute kernels
- AutoRound kernels
radiance_autoround_kernels.h - Drafter W4A16
sly/patch_w4a16_tiles.py - Escha kernels
radiance_escha.py
Attention and state
- Decode attention
sly/radiance_attn_decode.py - Drafter attention
sly/radiance_attn_drafter.py - Gated delta net
radiance_gdn.py - KV cache
Other components
- API user
Connections 20
- API user → vLLM serving: sends request
README.md - vLLM serving → API user: returns tokens
README.md - vLLM serving → DFlash2 drafter: drafts tokens
README.md - vLLM serving → Draft verification: verifies draft
README.md - vLLM serving → Prompt lookup: looks up prompts
README.md - DFlash2 drafter → Draft head: uses drafter
- vLLM serving → MXFP4 dispatch: dispatches GEMM
README.md - vLLM serving → NVFP4 support: loads checkpoint
README.md - vLLM serving → ParoQuant int4: loads checkpoint
README.md - vLLM serving → ParoQuant MXFP4: loads checkpoint
README.md - Quant registration → ParoQuant int4: registers config
sly/quant/radiance_quant_plugins.py - Quant registration → ParoQuant MXFP4: registers config
sly/quant/radiance_quant_plugins.py - ParoQuant MXFP4 → MXFP4 dispatch: uses GEMM
paroquant/radiance_paroquant_mxfp4.py - DFlash2 drafter → Drafter W4A16: uses tiled GEMM
sly/patch_w4a16_tiles.py - vLLM serving → AutoRound kernels: runs quantized GEMM
- vLLM serving → Escha kernels: loads checkpoint
README.md - vLLM serving → Decode attention: tunes verification
README.md - vLLM serving → Drafter attention: tunes draft attention
README.md - vLLM serving → Gated delta net: runs GDN kernels
README.md - vLLM serving → KV cache: reads and writes
README.md