How hicann/cann-recipes-infer works
CANN-RECIPES-INFER is a collection of Ascend NPU inference implementations and optimization recipes, not a single application. A typical user selects a model example, prepares its weights and environment, then runs inference through a native PyTorch-based implementation or an external framework integration. The tree and README document many model families and optimization modes; sampled source confirms parts of the executor, online serving, KV transfer, and selected multimodal pipelines. Wiring among unsampled model implementations is therefore left general rather than inferred.
Main entry points: execution_engine.py Execution engine · server.py Online server · online_inference.py Online inference · model_worker.py Model workers
Components 23
Inference entrypoints
- Offline inference
executor/offline/offline_inference.py - Online server
executor/online/server.py - Request routing
executor/online/router.py - Online inference
executor/online/online_inference.py - Batch scheduling
executor/core/scheduler/scheduler.py
Native runtime
- Execution engine
executor/core/engine/execution_engine.py - Model workers
executor/core/model_worker/model_worker.py - Model runner
executor/model_runner.py - Weight loading
executor/model_loader/default_loader.py - Paged KV cache
executor/core/kv_cache/kv_cache_manager.py - KV transfer
executor/online/kv_transfer/transfer_manager.py
Model recipes
- LLM recipes
models/deepseek_v4/models/modeling_deepseek.py - Vision-language
models/deepseek_v4_1/models/modeling_deepseek_vision.py - HSTU ranking
models/hstu/model/inference_ranking_gr.py
Multimodal generation
- Image generation
models/hunyuan-image-3.0/adaptor_patches/hunyuan_image_3_pipeline.py - Video generation
models/hunyuan-video/hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py - Audio-video generation
models/davinci-magihuman/inference/pipeline/pipeline.py
Framework integrations
- vLLM adapters
integration/vllm/ling-3.0-flash/npu_patch/bailing_v3_monkey_patch.py - SGLang recipes
integration/sglang/dsv4-flash-single-npu-moe-offload/scripts/tools/batch_convert_mxfp4_layers_mp.py - Ascend custom runtime
integration/custom/deepseek-v3-ascend310p/src/cpp/mlguider_llm_ascend/main_inference.cpp
Other components
- Developer
- Model weights
- Ascend NPU
Connections 23
- Developer → Offline inference: runs offline
- Developer → Online server: submits requests
- Online server → Request routing: dispatches requests
- Online server → Online inference: submits work
- Online inference → Batch scheduling: schedules batches
- Offline inference → Execution engine: runs inference
- Online inference → Execution engine: runs inference
- Execution engine → Model workers: invokes worker
- Model workers → Model runner: executes model
- Model workers → Weight loading: loads weights
- Model weights → Weight loading: provides weights
- Execution engine → Paged KV cache: allocates cache
- KV transfer → Ascend NPU: transfers KV
- Model runner → Ascend NPU: computes on
- LLM recipes → Ascend NPU: runs on
- Vision-language → Ascend NPU: runs on
- HSTU ranking → Ascend NPU: runs on
- Image generation → Ascend NPU: generates images
- Video generation → Ascend NPU: generates video
- Audio-video generation → Ascend NPU: generates media
- vLLM adapters → Ascend NPU: enables Ascend
- SGLang recipes → Ascend NPU: enables Ascend
- Ascend custom runtime → Ascend NPU: runs inference