# axrshz/hayate architecture

> Architecture diagram of the GitHub repository axrshz/hayate, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/axrshz/hayate
- Repository: https://github.com/axrshz/hayate

## Overview

Hayate is a CUDA inference engine for Qwen3. Its main workflow accepts prompts, prepares and schedules requests, prefills and decodes them in batches, maintains per-request KV state, and returns generated text. The model path comprises Qwen transformer blocks, attention, feed-forward layers, and RoPE; weight loading and optional prefix reuse support inference. The README also documents compilation and a workload benchmark. Evidence is strongest for the excerpted engine, model, cache, and utility code; links to tokenizer behavior and some unsampled runtime details are omitted where their calls are not shown.

Main entry points: [`hayate/engine/engine.py`](https://github.com/axrshz/hayate/blob/main/hayate/engine/engine.py) (Inference engine), [`hayate/model/block.py`](https://github.com/axrshz/hayate/blob/main/hayate/model/block.py) (Transformer block), [`hayate/engine/request.py`](https://github.com/axrshz/hayate/blob/main/hayate/engine/request.py) (Generation request), [`hayate/engine/scheduler.py`](https://github.com/axrshz/hayate/blob/main/hayate/engine/scheduler.py) (Batch scheduler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Inference runtime"]
  node_engine["Inference engine<br/>[engine.py]"]
  node_request["Generation request<br/>[request.py]"]
  node_scheduler["Batch scheduler<br/>[scheduler.py]"]
  node_sampler["Greedy sampler<br/>[sampler.py]"]
  node_tokenizer["Tokenizer<br/>[engine.py]"]
end

subgraph group_model["Qwen model"]
  node_qwen["Qwen3 model<br/>[qwen.py]"]
  node_block["Transformer block<br/>[block.py]"]
  node_attention["Grouped attention<br/>[attention.py]"]
  node_feedforward["Feed-forward layer<br/>[linear.py]"]
  node_rope["RoPE operations<br/>[rope.py]"]
end

subgraph group_state["Cache state"]
  node_cache[("KV cache<br/>[cache.py]")]
  node_cacheops["Batch cache ops<br/>[cache_ops.py]"]
  node_prefix[("Prefix cache<br/>[prefix_cache.py]")]
end

subgraph group_loading["Model loading"]
  node_weights["Weight loader<br/>[utils.py]"]
  node_hub["Hugging Face Hub"]
end

subgraph group_benchmark["Benchmarking"]
  node_benchmark["Workload benchmark<br/>[benchmark.py]"]
end

node_caller(("Application"))

node_caller -->|"requests generation"| node_engine
node_engine -->|"creates requests"| node_request
node_engine -->|"queues and ticks"| node_scheduler
node_scheduler -->|"admits requests"| node_request
node_engine -->|"samples tokens"| node_sampler
node_engine -.->|"tokenizes prompts"| node_tokenizer
node_engine -->|"runs forward pass"| node_qwen
node_qwen -->|"runs layers"| node_block
node_block -->|"computes attention"| node_attention
node_block -->|"computes MLP"| node_feedforward
node_attention -->|"applies RoPE"| node_rope
node_engine -->|"gathers and scatters"| node_cacheops
node_cacheops -->|"updates request caches"| node_request
node_engine -->|"creates and slices"| node_cache
node_engine -->|"looks up and stores"| node_prefix
node_prefix -->|"slices cache entries"| node_cache
node_engine -->|"loads model weights"| node_weights
node_weights -->|"downloads snapshot"| node_hub
node_caller -->|"runs workload"| node_benchmark

click node_engine "https://github.com/axrshz/hayate/blob/main/hayate/engine/engine.py"
click node_request "https://github.com/axrshz/hayate/blob/main/hayate/engine/request.py"
click node_scheduler "https://github.com/axrshz/hayate/blob/main/hayate/engine/scheduler.py"
click node_sampler "https://github.com/axrshz/hayate/blob/main/hayate/engine/sampler.py"
click node_tokenizer "https://github.com/axrshz/hayate/blob/main/hayate/engine/engine.py"
click node_qwen "https://github.com/axrshz/hayate/blob/main/hayate/model/qwen.py"
click node_block "https://github.com/axrshz/hayate/blob/main/hayate/model/block.py"
click node_attention "https://github.com/axrshz/hayate/blob/main/hayate/model/attention.py"
click node_feedforward "https://github.com/axrshz/hayate/blob/main/hayate/model/linear.py"
click node_rope "https://github.com/axrshz/hayate/blob/main/hayate/model/rope.py"
click node_cache "https://github.com/axrshz/hayate/blob/main/hayate/model/cache.py"
click node_cacheops "https://github.com/axrshz/hayate/blob/main/hayate/engine/cache_ops.py"
click node_prefix "https://github.com/axrshz/hayate/blob/main/hayate/engine/prefix_cache.py"
click node_weights "https://github.com/axrshz/hayate/blob/main/hayate/utils.py"
click node_benchmark "https://github.com/axrshz/hayate/blob/main/benchmark.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_engine,node_request,node_scheduler,node_sampler,node_tokenizer toneBlue
class node_qwen,node_block,node_attention,node_feedforward,node_rope toneAmber
class node_cache,node_cacheops,node_prefix toneMint
class node_weights,node_hub toneRose
class node_benchmark,node_caller toneIndigo
```

## Components

### Inference runtime

- [Inference engine](https://github.com/axrshz/hayate/blob/main/hayate/engine/engine.py): `hayate/engine/engine.py`
- [Generation request](https://github.com/axrshz/hayate/blob/main/hayate/engine/request.py): `hayate/engine/request.py`
- [Batch scheduler](https://github.com/axrshz/hayate/blob/main/hayate/engine/scheduler.py): `hayate/engine/scheduler.py`
- [Greedy sampler](https://github.com/axrshz/hayate/blob/main/hayate/engine/sampler.py): `hayate/engine/sampler.py`
- [Tokenizer](https://github.com/axrshz/hayate/blob/main/hayate/engine/engine.py): `hayate/engine/engine.py`

### Qwen model

- [Qwen3 model](https://github.com/axrshz/hayate/blob/main/hayate/model/qwen.py): `hayate/model/qwen.py`
- [Transformer block](https://github.com/axrshz/hayate/blob/main/hayate/model/block.py): `hayate/model/block.py`
- [Grouped attention](https://github.com/axrshz/hayate/blob/main/hayate/model/attention.py): `hayate/model/attention.py`
- [Feed-forward layer](https://github.com/axrshz/hayate/blob/main/hayate/model/linear.py): `hayate/model/linear.py`
- [RoPE operations](https://github.com/axrshz/hayate/blob/main/hayate/model/rope.py): `hayate/model/rope.py`

### Cache state

- [KV cache](https://github.com/axrshz/hayate/blob/main/hayate/model/cache.py): `hayate/model/cache.py`
- [Batch cache ops](https://github.com/axrshz/hayate/blob/main/hayate/engine/cache_ops.py): `hayate/engine/cache_ops.py`
- [Prefix cache](https://github.com/axrshz/hayate/blob/main/hayate/engine/prefix_cache.py): `hayate/engine/prefix_cache.py`

### Model loading

- [Weight loader](https://github.com/axrshz/hayate/blob/main/hayate/utils.py): `hayate/utils.py`
- **Hugging Face Hub**

### Benchmarking

- [Workload benchmark](https://github.com/axrshz/hayate/blob/main/benchmark.py): `benchmark.py`

### Other components

- **Application**

## Connections

- Application → Inference engine: requests generation (evidence: `README.md`)
- Inference engine → Generation request: creates requests (evidence: `hayate/engine/engine.py`)
- Inference engine → Batch scheduler: queues and ticks (evidence: `hayate/engine/engine.py`)
- Batch scheduler → Generation request: admits requests (evidence: `hayate/engine/scheduler.py`)
- Inference engine → Greedy sampler: samples tokens (evidence: `hayate/engine/engine.py`)
- Inference engine → Tokenizer: tokenizes prompts (evidence: `hayate/engine/engine.py`)
- Inference engine → Qwen3 model: runs forward pass (evidence: `hayate/engine/engine.py`)
- Qwen3 model → Transformer block: runs layers (evidence: `hayate/model/qwen.py`)
- Transformer block → Grouped attention: computes attention (evidence: `hayate/model/block.py`)
- Transformer block → Feed-forward layer: computes MLP (evidence: `hayate/model/block.py`)
- Grouped attention → RoPE operations: applies RoPE (evidence: `hayate/model/attention.py`)
- Inference engine → Batch cache ops: gathers and scatters (evidence: `hayate/engine/engine.py`)
- Batch cache ops → Generation request: updates request caches (evidence: `hayate/engine/cache_ops.py`)
- Inference engine → KV cache: creates and slices (evidence: `hayate/engine/engine.py`)
- Inference engine → Prefix cache: looks up and stores (evidence: `hayate/engine/engine.py`)
- Prefix cache → KV cache: slices cache entries (evidence: `hayate/engine/prefix_cache.py`)
- Inference engine → Weight loader: loads model weights (evidence: `hayate/engine/engine.py`)
- Weight loader → Hugging Face Hub: downloads snapshot (evidence: `hayate/utils.py`)
- Application → Workload benchmark: runs workload (evidence: `README.md`)
