How 1catai/1cat-vllm-gaudi works
1Cat-vLLM-Gaudi is a vLLM hardware plugin for Intel Gaudi, with DeepSeek V4.1 as a prominent optimization path. The principal workflow is request execution by the HPU worker, through model computation and persistent attention/pipeline state, to sampled output. The README also documents prepared-weight handling, calibration, native TPC operators, and graph/replay execution. The graph uses the supplied source excerpts and README; unsampled subsystem wiring is intentionally left out.
Main entry points: deepseek_v4 V4.1 model · hpu_worker.py HPU worker · entry_points.cpp Native operator glue · deepseek_v41_pipeline_batch.py PP batch transport
Components 21
Inference runtime
- HPU worker
vllm_gaudi/v1/worker/hpu_worker.py
DeepSeek V4.1
- PP batch transport
vllm_gaudi/v1/worker/deepseek_v41_pipeline_batch.py - V4.1 model
csrc/deepseek_v4 - KV and state
csrc/deepseek_v4/kernels - Prefix lookup
vllm_gaudi/ops/deepseek_v41_prefix_manager.py
Gaudi operators
- Attention kernels
csrc/deepseek_v4/kernels - Router top-6
csrc/deepseek_v4/pytorch/hpu_dsv41_router_logits_top6_pt2.cpp - Expert kernels
csrc/deepseek_v4/kernels - Native operator glue
csrc/deepseek_v4/host/entry_points.cpp
Weights and calibration
- Prepared weights
tools/prepare_deepseek_v4_engine.py - Scale calibration
calibration/step-0-detect-device.py - Scale measurement
calibration/step-2-measure-scales.py - Scale processing
calibration/step-3-postprocess-measure.py - Scale quantization
calibration/step-4-quantize-scales.py - Scale unification
calibration/step-5-unify_measurements.py - Scale expansion
calibration/step-6-expand-measurements.py
Other components
- Request caller
- vLLM engine
- Model checkpoint
- Gaudi HPU
- Sampled output
Connections 21
- Request caller → vLLM engine: submits request
- vLLM engine → HPU worker: dispatches work
- HPU worker → PP batch transport: runs batches
- PP batch transport → V4.1 model: transports activations
- V4.1 model → Attention kernels: computes attention
- V4.1 model → Router top-6: routes tokens
- V4.1 model → Expert kernels: computes experts
- V4.1 model → KV and state: reads and updates
- HPU worker → Prefix lookup: uses prefix lookup
- Router top-6 → Native operator glue: dispatches operator
- Attention kernels → Native operator glue: dispatches operator
- Expert kernels → Native operator glue: dispatches operator
- Native operator glue → Gaudi HPU: executes kernels
- Model checkpoint → Prepared weights: provides weights
- Prepared weights → V4.1 model: loads weights
- Scale calibration → Scale measurement: measures scales
- Scale measurement → Scale processing: processes measurements
- Scale processing → Scale quantization: quantizes scales
- Scale quantization → Scale unification: unifies scales
- Scale unification → Scale expansion: expands scales
- HPU worker → Sampled output: returns tokens