How axrshz/hayate works
Hayate is a CUDA inference engine for Qwen3. Its main workflow accepts prompts, prepares and schedules requests, prefills and decodes them in batches, maintains per-request KV state, and returns generated text. The model path comprises Qwen transformer blocks, attention, feed-forward layers, and RoPE; weight loading and optional prefix reuse support inference. The README also documents compilation and a workload benchmark. Evidence is strongest for the excerpted engine, model, cache, and utility code; links to tokenizer behavior and some unsampled runtime details are omitted where their calls are not shown.
Main entry points: engine.py Inference engine · block.py Transformer block · request.py Generation request · scheduler.py Batch scheduler
Components 17
Inference runtime
- Inference engine
hayate/engine/engine.py - Generation request
hayate/engine/request.py - Batch scheduler
hayate/engine/scheduler.py - Greedy sampler
hayate/engine/sampler.py - Tokenizer
hayate/engine/engine.py
Qwen model
- Qwen3 model
hayate/model/qwen.py - Transformer block
hayate/model/block.py - Grouped attention
hayate/model/attention.py - Feed-forward layer
hayate/model/linear.py - RoPE operations
hayate/model/rope.py
Cache state
- KV cache
hayate/model/cache.py - Batch cache ops
hayate/engine/cache_ops.py - Prefix cache
hayate/engine/prefix_cache.py
Model loading
- Weight loader
hayate/utils.py - Hugging Face Hub
Benchmarking
- Workload benchmark
benchmark.py
Other components
- Application
Connections 19
- Application → Inference engine: requests generation
README.md - Inference engine → Generation request: creates requests
hayate/engine/engine.py - Inference engine → Batch scheduler: queues and ticks
hayate/engine/engine.py - Batch scheduler → Generation request: admits requests
hayate/engine/scheduler.py - Inference engine → Greedy sampler: samples tokens
hayate/engine/engine.py - Inference engine → Tokenizer: tokenizes prompts
hayate/engine/engine.py - Inference engine → Qwen3 model: runs forward pass
hayate/engine/engine.py - Qwen3 model → Transformer block: runs layers
hayate/model/qwen.py - Transformer block → Grouped attention: computes attention
hayate/model/block.py - Transformer block → Feed-forward layer: computes MLP
hayate/model/block.py - Grouped attention → RoPE operations: applies RoPE
hayate/model/attention.py - Inference engine → Batch cache ops: gathers and scatters
hayate/engine/engine.py - Batch cache ops → Generation request: updates request caches
hayate/engine/cache_ops.py - Inference engine → KV cache: creates and slices
hayate/engine/engine.py - Inference engine → Prefix cache: looks up and stores
hayate/engine/engine.py - Prefix cache → KV cache: slices cache entries
hayate/engine/prefix_cache.py - Inference engine → Weight loader: loads model weights
hayate/engine/engine.py - Weight loader → Hugging Face Hub: downloads snapshot
hayate/utils.py - Application → Workload benchmark: runs workload
README.md