How syv-ai/hyperqwen works
HyperQwen packages a patched vLLM serving setup for large Qwen models, plus model-preparation tools. The main workflow is choosing single-user or batch operation, preparing model weights when needed, then sending requests to an OpenAI-compatible API. The product also offers speculative decoding and KVarN long-context KV-cache compression; preparation includes draft-vocabulary and MTP quantization. The README documents these capabilities, while sampled source gives detail on selected preparation and KVarN internals, not the full serving implementation.
Main entry points: PATCHES.md Patched vLLM · kvarn_attn.py KVarN backend · patches Speculative decoding · export_mtp.py MTP export
Components 18
Serving modes
- Single mode
single-user - Batch mode
batch
Inference runtime
- OpenAI API
patches/serve-404-served-names.patch - Patched vLLM
PATCHES.md - Speculative decoding
patches - MTP drafter
prepare/quant_mtp.py - DFlash2 drafter
drafter
Model preparation
- Model preparation
prepare/README.md - Head quantization
prepare/quant_heads_stream.py - Draft vocabulary
prepare/build_draft_vocab.py - MTP export
drafter/export_mtp.py - MTP training
drafter/train_mtp.py
Long-context cache
- KVarN backend
kvarn/files/vllm/v1/attention/backends/kvarn_attn.py - Variance normalization
kvarn/files/vllm/model_executor/layers/quantization/kvarn/sinkhorn.py - KV tile storage
kvarn/files/vllm/v1/attention/ops/kvarn_store.py - Fused decode
kvarn/files/vllm/v1/attention/ops/triton_kvarn_decode.py
Other components
- API client
- Model checkpoint
Connections 19
- API client → Single mode: selects mode
- API client → Batch mode: selects mode
- Single mode → Patched vLLM: starts runtime
- Batch mode → Patched vLLM: starts runtime
- API client → OpenAI API: sends requests
- OpenAI API → Patched vLLM: dispatches requests
- Model preparation → Model checkpoint: reads weights
- Model preparation → Head quantization: runs quantization
- Head quantization → Draft vocabulary: enables draft head
- MTP training → MTP export: provides trained weights
- MTP export → MTP drafter: exports drafter
- MTP export → Draft vocabulary: builds draft head
- Patched vLLM → Speculative decoding: uses patches
- Speculative decoding → MTP drafter: configures speculation
- Speculative decoding → DFlash2 drafter: configures speculation
- Patched vLLM → KVarN backend: selects backend
- KVarN backend → Variance normalization: normalizes tiles
- KVarN backend → KV tile storage: stores KV tiles
- KVarN backend → Fused decode: decodes cache