# 1catai/1cat-vllm-gaudi architecture

> Architecture diagram of the GitHub repository 1catai/1cat-vllm-gaudi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/1catai/1cat-vllm-gaudi
- Repository: https://github.com/1catai/1cat-vllm-gaudi

## Overview

1Cat-vLLM-Gaudi is a vLLM hardware plugin for Intel Gaudi, with DeepSeek V4.1 as a prominent optimization path. The principal workflow is request execution by the HPU worker, through model computation and persistent attention/pipeline state, to sampled output. The README also documents prepared-weight handling, calibration, native TPC operators, and graph/replay execution. The graph uses the supplied source excerpts and README; unsampled subsystem wiring is intentionally left out.

Main entry points: [`csrc/deepseek_v4`](https://github.com/1catai/1cat-vllm-gaudi/tree/main/csrc/deepseek_v4) (V4.1 model), [`vllm_gaudi/v1/worker/hpu_worker.py`](https://github.com/1catai/1cat-vllm-gaudi/blob/main/vllm_gaudi/v1/worker/hpu_worker.py) (HPU worker), [`csrc/deepseek_v4/host/entry_points.cpp`](https://github.com/1catai/1cat-vllm-gaudi/blob/main/csrc/deepseek_v4/host/entry_points.cpp) (Native operator glue), [`vllm_gaudi/v1/worker/deepseek_v41_pipeline_batch.py`](https://github.com/1catai/1cat-vllm-gaudi/blob/main/vllm_gaudi/v1/worker/deepseek_v41_pipeline_batch.py) (PP batch transport)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Inference runtime"]
  node_worker["HPU worker<br/>[hpu_worker.py]"]
end

subgraph group_deepseek["DeepSeek V4.1"]
  node_pipeline["PP batch transport"]
  node_model["V4.1 model"]
  node_kvstate[("KV and state")]
  node_prefix["Prefix lookup"]
end

subgraph group_native["Gaudi operators"]
  node_attention["Attention kernels"]
  node_routing["Router top-6"]
  node_experts["Expert kernels"]
  node_operators["Native operator glue<br/>[entry_points.cpp]"]
end

subgraph group_weights["Weights and calibration"]
  node_prepared["Prepared weights"]
  node_calibrate["Scale calibration"]
  node_measure["Scale measurement"]
  node_postprocess["Scale processing"]
  node_quantize["Scale quantization"]
  node_unify["Scale unification"]
  node_expand["Scale expansion"]
end

node_caller(("Request caller"))
node_engine["vLLM engine"]
node_checkpoint["Model checkpoint"]
node_hpu(("Gaudi HPU"))
node_sample["Sampled output"]

node_caller -->|"submits request"| node_engine
node_engine -->|"dispatches work"| node_worker
node_worker -->|"runs batches"| node_pipeline
node_pipeline -->|"transports activations"| node_model
node_model -->|"computes attention"| node_attention
node_model -->|"routes tokens"| node_routing
node_model -->|"computes experts"| node_experts
node_model -->|"reads and updates"| node_kvstate
node_worker -.->|"uses prefix lookup"| node_prefix
node_routing -->|"dispatches operator"| node_operators
node_attention -->|"dispatches operator"| node_operators
node_experts -->|"dispatches operator"| node_operators
node_operators -->|"executes kernels"| node_hpu
node_checkpoint -->|"provides weights"| node_prepared
node_prepared -->|"loads weights"| node_model
node_calibrate -->|"measures scales"| node_measure
node_measure -->|"processes measurements"| node_postprocess
node_postprocess -->|"quantizes scales"| node_quantize
node_quantize -->|"unifies scales"| node_unify
node_unify -->|"expands scales"| node_expand
node_worker -->|"returns tokens"| node_sample

click node_worker "https://github.com/1catai/1cat-vllm-gaudi/blob/main/vllm_gaudi/v1/worker/hpu_worker.py"
click node_pipeline "https://github.com/1catai/1cat-vllm-gaudi/blob/main/vllm_gaudi/v1/worker/deepseek_v41_pipeline_batch.py"
click node_model "https://github.com/1catai/1cat-vllm-gaudi/tree/main/csrc/deepseek_v4"
click node_attention "https://github.com/1catai/1cat-vllm-gaudi/tree/main/csrc/deepseek_v4/kernels"
click node_routing "https://github.com/1catai/1cat-vllm-gaudi/blob/main/csrc/deepseek_v4/pytorch/hpu_dsv41_router_logits_top6_pt2.cpp"
click node_experts "https://github.com/1catai/1cat-vllm-gaudi/tree/main/csrc/deepseek_v4/kernels"
click node_kvstate "https://github.com/1catai/1cat-vllm-gaudi/tree/main/csrc/deepseek_v4/kernels"
click node_prefix "https://github.com/1catai/1cat-vllm-gaudi/blob/main/vllm_gaudi/ops/deepseek_v41_prefix_manager.py"
click node_operators "https://github.com/1catai/1cat-vllm-gaudi/blob/main/csrc/deepseek_v4/host/entry_points.cpp"
click node_prepared "https://github.com/1catai/1cat-vllm-gaudi/blob/main/tools/prepare_deepseek_v4_engine.py"
click node_calibrate "https://github.com/1catai/1cat-vllm-gaudi/blob/main/calibration/step-0-detect-device.py"
click node_measure "https://github.com/1catai/1cat-vllm-gaudi/blob/main/calibration/step-2-measure-scales.py"
click node_postprocess "https://github.com/1catai/1cat-vllm-gaudi/blob/main/calibration/step-3-postprocess-measure.py"
click node_quantize "https://github.com/1catai/1cat-vllm-gaudi/blob/main/calibration/step-4-quantize-scales.py"
click node_unify "https://github.com/1catai/1cat-vllm-gaudi/blob/main/calibration/step-5-unify_measurements.py"
click node_expand "https://github.com/1catai/1cat-vllm-gaudi/blob/main/calibration/step-6-expand-measurements.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_worker toneBlue
class node_pipeline,node_model,node_kvstate,node_prefix toneAmber
class node_attention,node_routing,node_experts,node_operators,node_caller toneMint
class node_prepared,node_calibrate,node_measure,node_postprocess,node_quantize,node_unify,node_expand toneRose
class node_engine,node_checkpoint,node_hpu,node_sample toneIndigo
```

## Components

### Inference runtime

- [HPU worker](https://github.com/1catai/1cat-vllm-gaudi/blob/main/vllm_gaudi/v1/worker/hpu_worker.py): `vllm_gaudi/v1/worker/hpu_worker.py`

### DeepSeek V4.1

- [PP batch transport](https://github.com/1catai/1cat-vllm-gaudi/blob/main/vllm_gaudi/v1/worker/deepseek_v41_pipeline_batch.py): `vllm_gaudi/v1/worker/deepseek_v41_pipeline_batch.py`
- [V4.1 model](https://github.com/1catai/1cat-vllm-gaudi/tree/main/csrc/deepseek_v4): `csrc/deepseek_v4`
- [KV and state](https://github.com/1catai/1cat-vllm-gaudi/tree/main/csrc/deepseek_v4/kernels): `csrc/deepseek_v4/kernels`
- [Prefix lookup](https://github.com/1catai/1cat-vllm-gaudi/blob/main/vllm_gaudi/ops/deepseek_v41_prefix_manager.py): `vllm_gaudi/ops/deepseek_v41_prefix_manager.py`

### Gaudi operators

- [Attention kernels](https://github.com/1catai/1cat-vllm-gaudi/tree/main/csrc/deepseek_v4/kernels): `csrc/deepseek_v4/kernels`
- [Router top-6](https://github.com/1catai/1cat-vllm-gaudi/blob/main/csrc/deepseek_v4/pytorch/hpu_dsv41_router_logits_top6_pt2.cpp): `csrc/deepseek_v4/pytorch/hpu_dsv41_router_logits_top6_pt2.cpp`
- [Expert kernels](https://github.com/1catai/1cat-vllm-gaudi/tree/main/csrc/deepseek_v4/kernels): `csrc/deepseek_v4/kernels`
- [Native operator glue](https://github.com/1catai/1cat-vllm-gaudi/blob/main/csrc/deepseek_v4/host/entry_points.cpp): `csrc/deepseek_v4/host/entry_points.cpp`

### Weights and calibration

- [Prepared weights](https://github.com/1catai/1cat-vllm-gaudi/blob/main/tools/prepare_deepseek_v4_engine.py): `tools/prepare_deepseek_v4_engine.py`
- [Scale calibration](https://github.com/1catai/1cat-vllm-gaudi/blob/main/calibration/step-0-detect-device.py): `calibration/step-0-detect-device.py`
- [Scale measurement](https://github.com/1catai/1cat-vllm-gaudi/blob/main/calibration/step-2-measure-scales.py): `calibration/step-2-measure-scales.py`
- [Scale processing](https://github.com/1catai/1cat-vllm-gaudi/blob/main/calibration/step-3-postprocess-measure.py): `calibration/step-3-postprocess-measure.py`
- [Scale quantization](https://github.com/1catai/1cat-vllm-gaudi/blob/main/calibration/step-4-quantize-scales.py): `calibration/step-4-quantize-scales.py`
- [Scale unification](https://github.com/1catai/1cat-vllm-gaudi/blob/main/calibration/step-5-unify_measurements.py): `calibration/step-5-unify_measurements.py`
- [Scale expansion](https://github.com/1catai/1cat-vllm-gaudi/blob/main/calibration/step-6-expand-measurements.py): `calibration/step-6-expand-measurements.py`

### Other components

- **Request caller**
- **vLLM engine**
- **Model checkpoint**
- **Gaudi HPU**
- **Sampled output**

## Connections

- Request caller → vLLM engine: submits request
- vLLM engine → HPU worker: dispatches work
- HPU worker → PP batch transport: runs batches
- PP batch transport → V4.1 model: transports activations
- V4.1 model → Attention kernels: computes attention
- V4.1 model → Router top-6: routes tokens
- V4.1 model → Expert kernels: computes experts
- V4.1 model → KV and state: reads and updates
- HPU worker → Prefix lookup: uses prefix lookup
- Router top-6 → Native operator glue: dispatches operator
- Attention kernels → Native operator glue: dispatches operator
- Expert kernels → Native operator glue: dispatches operator
- Native operator glue → Gaudi HPU: executes kernels
- Model checkpoint → Prepared weights: provides weights
- Prepared weights → V4.1 model: loads weights
- Scale calibration → Scale measurement: measures scales
- Scale measurement → Scale processing: processes measurements
- Scale processing → Scale quantization: quantizes scales
- Scale quantization → Scale unification: unifies scales
- Scale unification → Scale expansion: expands scales
- HPU worker → Sampled output: returns tokens