# hicann/cann-recipes-infer architecture

> Architecture diagram of the GitHub repository hicann/cann-recipes-infer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hicann/cann-recipes-infer
- Repository: https://github.com/hicann/cann-recipes-infer

## Overview

CANN-RECIPES-INFER is a collection of Ascend NPU inference implementations and optimization recipes, not a single application. A typical user selects a model example, prepares its weights and environment, then runs inference through a native PyTorch-based implementation or an external framework integration. The tree and README document many model families and optimization modes; sampled source confirms parts of the executor, online serving, KV transfer, and selected multimodal pipelines. Wiring among unsampled model implementations is therefore left general rather than inferred.

Main entry points: [`executor/core/engine/execution_engine.py`](https://github.com/hicann/cann-recipes-infer/blob/master/executor/core/engine/execution_engine.py) (Execution engine), [`executor/online/server.py`](https://github.com/hicann/cann-recipes-infer/blob/master/executor/online/server.py) (Online server), [`executor/online/online_inference.py`](https://github.com/hicann/cann-recipes-infer/blob/master/executor/online/online_inference.py) (Online inference), [`executor/core/model_worker/model_worker.py`](https://github.com/hicann/cann-recipes-infer/blob/master/executor/core/model_worker/model_worker.py) (Model workers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Inference entrypoints"]
  node_offline["Offline inference"]
  node_online["Online server<br/>[server.py]"]
  node_router["Request routing<br/>[router.py]"]
  node_online_engine["Online inference"]
  node_scheduler["Batch scheduling<br/>[scheduler.py]"]
end

subgraph group_runtime["Native runtime"]
  node_execution["Execution engine"]
  node_worker["Model workers<br/>[model_worker.py]"]
  node_runner["Model runner<br/>[model_runner.py]"]
  node_loader["Weight loading<br/>[default_loader.py]"]
  node_kv[("Paged KV cache")]
  node_transfer["KV transfer"]
end

subgraph group_models["Model recipes"]
  node_llm["LLM recipes"]
  node_vision_llm["Vision-language"]
  node_hstu["HSTU ranking"]
end

subgraph group_multimodal["Multimodal generation"]
  node_image["Image generation"]
  node_video["Video generation"]
  node_magihuman["Audio-video generation<br/>[pipeline.py]"]
end

subgraph group_integrations["Framework integrations"]
  node_vllm["vLLM adapters"]
  node_sglang["SGLang recipes"]
  node_custom["Ascend custom runtime<br/>[main_inference.cpp]"]
end

node_developer(("Developer"))
node_artifacts[("Model weights")]
node_npu{{"Ascend NPU"}}

node_developer -->|"runs offline"| node_offline
node_developer -->|"submits requests"| node_online
node_online -->|"dispatches requests"| node_router
node_online -->|"submits work"| node_online_engine
node_online_engine -->|"schedules batches"| node_scheduler
node_offline -->|"runs inference"| node_execution
node_online_engine -->|"runs inference"| node_execution
node_execution -->|"invokes worker"| node_worker
node_worker -->|"executes model"| node_runner
node_worker -->|"loads weights"| node_loader
node_artifacts -->|"provides weights"| node_loader
node_execution -->|"allocates cache"| node_kv
node_transfer -.->|"transfers KV"| node_npu
node_runner -->|"computes on"| node_npu
node_llm -->|"runs on"| node_npu
node_vision_llm -->|"runs on"| node_npu
node_hstu -->|"runs on"| node_npu
node_image -->|"generates images"| node_npu
node_video -->|"generates video"| node_npu
node_magihuman -->|"generates media"| node_npu
node_vllm -.->|"enables Ascend"| node_npu
node_sglang -.->|"enables Ascend"| node_npu
node_custom -->|"runs inference"| node_npu

click node_offline "https://github.com/hicann/cann-recipes-infer/blob/master/executor/offline/offline_inference.py"
click node_online "https://github.com/hicann/cann-recipes-infer/blob/master/executor/online/server.py"
click node_router "https://github.com/hicann/cann-recipes-infer/blob/master/executor/online/router.py"
click node_online_engine "https://github.com/hicann/cann-recipes-infer/blob/master/executor/online/online_inference.py"
click node_scheduler "https://github.com/hicann/cann-recipes-infer/blob/master/executor/core/scheduler/scheduler.py"
click node_execution "https://github.com/hicann/cann-recipes-infer/blob/master/executor/core/engine/execution_engine.py"
click node_worker "https://github.com/hicann/cann-recipes-infer/blob/master/executor/core/model_worker/model_worker.py"
click node_runner "https://github.com/hicann/cann-recipes-infer/blob/master/executor/model_runner.py"
click node_loader "https://github.com/hicann/cann-recipes-infer/blob/master/executor/model_loader/default_loader.py"
click node_kv "https://github.com/hicann/cann-recipes-infer/blob/master/executor/core/kv_cache/kv_cache_manager.py"
click node_transfer "https://github.com/hicann/cann-recipes-infer/blob/master/executor/online/kv_transfer/transfer_manager.py"
click node_llm "https://github.com/hicann/cann-recipes-infer/blob/master/models/deepseek_v4/models/modeling_deepseek.py"
click node_vision_llm "https://github.com/hicann/cann-recipes-infer/blob/master/models/deepseek_v4_1/models/modeling_deepseek_vision.py"
click node_hstu "https://github.com/hicann/cann-recipes-infer/blob/master/models/hstu/model/inference_ranking_gr.py"
click node_image "https://github.com/hicann/cann-recipes-infer/blob/master/models/hunyuan-image-3.0/adaptor_patches/hunyuan_image_3_pipeline.py"
click node_video "https://github.com/hicann/cann-recipes-infer/blob/master/models/hunyuan-video/hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py"
click node_magihuman "https://github.com/hicann/cann-recipes-infer/blob/master/models/davinci-magihuman/inference/pipeline/pipeline.py"
click node_vllm "https://github.com/hicann/cann-recipes-infer/blob/master/integration/vllm/ling-3.0-flash/npu_patch/bailing_v3_monkey_patch.py"
click node_sglang "https://github.com/hicann/cann-recipes-infer/blob/master/integration/sglang/dsv4-flash-single-npu-moe-offload/scripts/tools/batch_convert_mxfp4_layers_mp.py"
click node_custom "https://github.com/hicann/cann-recipes-infer/blob/master/integration/custom/deepseek-v3-ascend310p/src/cpp/mlguider_llm_ascend/main_inference.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_offline,node_online,node_router,node_online_engine,node_scheduler toneBlue
class node_execution,node_worker,node_runner,node_loader,node_kv,node_transfer,node_artifacts toneAmber
class node_llm,node_vision_llm,node_hstu toneMint
class node_image,node_video,node_magihuman toneRose
class node_vllm,node_sglang,node_custom,node_developer,node_npu toneIndigo
```

## Components

### Inference entrypoints

- [Offline inference](https://github.com/hicann/cann-recipes-infer/blob/master/executor/offline/offline_inference.py): `executor/offline/offline_inference.py`
- [Online server](https://github.com/hicann/cann-recipes-infer/blob/master/executor/online/server.py): `executor/online/server.py`
- [Request routing](https://github.com/hicann/cann-recipes-infer/blob/master/executor/online/router.py): `executor/online/router.py`
- [Online inference](https://github.com/hicann/cann-recipes-infer/blob/master/executor/online/online_inference.py): `executor/online/online_inference.py`
- [Batch scheduling](https://github.com/hicann/cann-recipes-infer/blob/master/executor/core/scheduler/scheduler.py): `executor/core/scheduler/scheduler.py`

### Native runtime

- [Execution engine](https://github.com/hicann/cann-recipes-infer/blob/master/executor/core/engine/execution_engine.py): `executor/core/engine/execution_engine.py`
- [Model workers](https://github.com/hicann/cann-recipes-infer/blob/master/executor/core/model_worker/model_worker.py): `executor/core/model_worker/model_worker.py`
- [Model runner](https://github.com/hicann/cann-recipes-infer/blob/master/executor/model_runner.py): `executor/model_runner.py`
- [Weight loading](https://github.com/hicann/cann-recipes-infer/blob/master/executor/model_loader/default_loader.py): `executor/model_loader/default_loader.py`
- [Paged KV cache](https://github.com/hicann/cann-recipes-infer/blob/master/executor/core/kv_cache/kv_cache_manager.py): `executor/core/kv_cache/kv_cache_manager.py`
- [KV transfer](https://github.com/hicann/cann-recipes-infer/blob/master/executor/online/kv_transfer/transfer_manager.py): `executor/online/kv_transfer/transfer_manager.py`

### Model recipes

- [LLM recipes](https://github.com/hicann/cann-recipes-infer/blob/master/models/deepseek_v4/models/modeling_deepseek.py): `models/deepseek_v4/models/modeling_deepseek.py`
- [Vision-language](https://github.com/hicann/cann-recipes-infer/blob/master/models/deepseek_v4_1/models/modeling_deepseek_vision.py): `models/deepseek_v4_1/models/modeling_deepseek_vision.py`
- [HSTU ranking](https://github.com/hicann/cann-recipes-infer/blob/master/models/hstu/model/inference_ranking_gr.py): `models/hstu/model/inference_ranking_gr.py`

### Multimodal generation

- [Image generation](https://github.com/hicann/cann-recipes-infer/blob/master/models/hunyuan-image-3.0/adaptor_patches/hunyuan_image_3_pipeline.py): `models/hunyuan-image-3.0/adaptor_patches/hunyuan_image_3_pipeline.py`
- [Video generation](https://github.com/hicann/cann-recipes-infer/blob/master/models/hunyuan-video/hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py): `models/hunyuan-video/hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py`
- [Audio-video generation](https://github.com/hicann/cann-recipes-infer/blob/master/models/davinci-magihuman/inference/pipeline/pipeline.py): `models/davinci-magihuman/inference/pipeline/pipeline.py`

### Framework integrations

- [vLLM adapters](https://github.com/hicann/cann-recipes-infer/blob/master/integration/vllm/ling-3.0-flash/npu_patch/bailing_v3_monkey_patch.py): `integration/vllm/ling-3.0-flash/npu_patch/bailing_v3_monkey_patch.py`
- [SGLang recipes](https://github.com/hicann/cann-recipes-infer/blob/master/integration/sglang/dsv4-flash-single-npu-moe-offload/scripts/tools/batch_convert_mxfp4_layers_mp.py): `integration/sglang/dsv4-flash-single-npu-moe-offload/scripts/tools/batch_convert_mxfp4_layers_mp.py`
- [Ascend custom runtime](https://github.com/hicann/cann-recipes-infer/blob/master/integration/custom/deepseek-v3-ascend310p/src/cpp/mlguider_llm_ascend/main_inference.cpp): `integration/custom/deepseek-v3-ascend310p/src/cpp/mlguider_llm_ascend/main_inference.cpp`

### Other components

- **Developer**
- **Model weights**
- **Ascend NPU**

## Connections

- Developer → Offline inference: runs offline
- Developer → Online server: submits requests
- Online server → Request routing: dispatches requests
- Online server → Online inference: submits work
- Online inference → Batch scheduling: schedules batches
- Offline inference → Execution engine: runs inference
- Online inference → Execution engine: runs inference
- Execution engine → Model workers: invokes worker
- Model workers → Model runner: executes model
- Model workers → Weight loading: loads weights
- Model weights → Weight loading: provides weights
- Execution engine → Paged KV cache: allocates cache
- KV transfer → Ascend NPU: transfers KV
- Model runner → Ascend NPU: computes on
- LLM recipes → Ascend NPU: runs on
- Vision-language → Ascend NPU: runs on
- HSTU ranking → Ascend NPU: runs on
- Image generation → Ascend NPU: generates images
- Video generation → Ascend NPU: generates video
- Audio-video generation → Ascend NPU: generates media
- vLLM adapters → Ascend NPU: enables Ascend
- SGLang recipes → Ascend NPU: enables Ascend
- Ascend custom runtime → Ascend NPU: runs inference