# xwqtju/cann-hccl architecture

> Architecture diagram of the GitHub repository xwqtju/cann-hccl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xwqtju/cann-hccl
- Repository: https://github.com/xwqtju/cann-hccl

## Overview

#### Purpose and Entry Points
cann-hccl is Huawei’s C++ collective-communication implementation for Ascend NPUs, covering single- and multi-node data/model-parallel operations. This repository supplies the communication framework and algorithm layers; the underlying communication platform is provided by the installed CANN toolkit.

Public consumer-facing API/types are under `inc/hccl/hccl.h` and `inc/hccl/hccl_types.h`. The build/package entry point is `build.sh`, with the root build graph in `CMakeLists.txt`.

#### Layered Execution Path
A collective call is routed through framework communicator management, algorithm/operator selection, resource preparation, and task execution:
1. Framework initializes the communication domain and communicator.
2. Topology/rank information and operator parameters drive algorithm selection.
3. Operators obtain a specialized collective executor from registries.
4. Executors compose topology-specific base algorithms and transport requirements.
5. Tasks are submitted using streams, buffers, queues/notifies, and platform-backed transport resources.

Key orchestration paths: `src/domain/collective_communication/framework/communicator/impl/hccl_communicator.cc`, `src/domain/collective_communication/algorithm/impl/hccl_impl.cc`, and `src/domain/collective_communication/algorithm/impl/hccl_pipeline.cc`.

#### Framework: Communicators and Topology
The framework owns communication-domain lifecycle and bridges application calls to algorithm execution. `HcclCommunicator` and transport management form the principal boundary to the CANN communication platform. Relevant code: `src/domain/collective_communication/framework/communicator/impl/hccl_communicator.cc`, `src/domain/collective_communication/framework/communicator/impl/resource_manager/transport_manager.cc`, and `src/domain/collective_communication/framework/op_base/src/op_base.cc`.

Topology discovery, rank-table parsing, whitelist handling, and topology exchange are centralized separately. These inputs determine rank relationships and topology-aware planning: `src/domain/collective_communication/framework/common/src/topo/topoinfo_detect.cc`, `src/domain/collective_communication/framework/common/src/topo/topoinfo_ranktableParser.cc`, and `src/domain/collective_communication/framework/common/src/topo/topoinfo_exchange_dispatcher.cc`.

#### Operator Selection and Algorithm Dispatch
Each collective has an operator adapter responsible for validation/planning integration, including AllReduce, AllGather, AllToAll, Broadcast, Reduce, ReduceScatter, Scatter, and point-to-point operations. Operators and executors are selected through registries, creating the main extensibility boundary for custom operations/algorithms.

Grounding paths: `src/domain/collective_communication/algorithm/impl/operator/coll_alg_operator.cc`, `src/domain/collective_communication/algorithm/impl/operator/all_reduce_operator.cc`, and `src/domain/collective_communication/algorithm/impl/coll_executor/registry/coll_alg_exec_registry.cc`.

#### Collective Executors
Operation-specific executors implement concrete plans for collective families and hardware/count variants. For example, AllReduce has ring, mesh, one-shot, pipeline, RDMA/AIV, and Ascend 310P-specific executors; comparable families exist for AllGather, Broadcast, ReduceScatter, AllToAll, Reduce, and Scatter.

Representative paths: `src/domain/collective_communication/algorithm/impl/coll_executor/coll_all_reduce/coll_all_reduce_executor.cc`, `src/domain/collective_communication/algorithm/impl/coll_executor/coll_all_to_all/coll_all_to_all_v_staged_executor.cc`, and `src/domain/collective_communication/algorithm/impl/coll_executor/coll_reduce_scatter/coll_reduce_scatter_ring_executor.cc`.

#### Base Topologies and Data Movement
Reusable base communicators calculate transport requirements and encode Mesh, Ring, point-to-point, Star, and recursive halving-doubling communication structures. Base executors implement the actual phased send/receive/reduce schedules, including direct, atomic, concurrent, and staged variants.

Key paths: `src/domain/collective_communication/algorithm/base/communicator/comm_factory.cc`, `src/domain/collective_communication/algorithm/base/communicator/calc_ring_transport_req.cc`, and `src/domain/collective_communication/algorithm/base/executor/all_reduce_ring.cc`.

#### Resource and Runtime Boundary
Algorithm execution depends on managed CCL buffers/workspace, sockets, streams, queue notifications, and asynchronous task loading. These are the material runtime boundary between planner/executor code and CANN’s NPU/transport services; this repository does not contain the platform implementation itself.

Relevant paths: `src/domain/collective_communication/algorithm/impl/resource_manager/ccl_buffer_manager.cc`, `src/domain/collective_communication/algorithm/impl/resource_manager/offload_stream_manager.cc`, and `src/domain/collective_communication/algorithm/impl/task/parallel_task_loader.cc`.

#### Build and External Dependencies
The project builds with CMake/GCC/Python on Linux and produces an installable CANN HCCL algorithm package. It requires a version-compatible CANN development toolkit and external nlohmann/json headers; runtime validation is performed by the separately supplied HCCL Test/MPI tooling. Build integration is defined in `CMakeLists.txt`, `cmake/config.cmake`, and `cmake/intf_pub_linux.cmake`.

Main entry points: [`src/domain/collective_communication/framework/communicator/impl/hccl_communicator.cc`](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/framework/communicator/impl/hccl_communicator.cc) (HcclCommunicator), [`src/domain/collective_communication/algorithm/base/executor/all_reduce_ring.cc`](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/base/executor/all_reduce_ring.cc) (Base algorithm executors), [`src/domain/collective_communication/algorithm/impl/hccl_impl.cc`](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/hccl_impl.cc) (Collective algorithm orchestration), [`src/domain/collective_communication/algorithm/impl/coll_executor/coll_all_reduce/coll_all_reduce_executor.cc`](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/coll_executor/coll_all_reduce/coll_all_reduce_executor.cc) (Collective executors)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Public & Build"]
  node_public_api["HCCL public API<br/>C++ API<br/>[hccl.h]"]
  node_build["CMake package build<br/>build entry<br/>[CMakeLists.txt]"]
end

subgraph group_framework["Communication Framework"]
  node_communicator["HcclCommunicator<br/>communicator lifecycle"]
  node_op_base["Operation base<br/>framework operation<br/>[op_base.cc]"]
  node_topology["Topology discovery<br/>rank topology<br/>[topoinfo_detect.cc]"]
  node_transport_manager["Transport manager<br/>transport lifecycle"]
end

subgraph group_algorithm["Algorithm Planning &amp; Execution"]
  node_algorithm_impl["Collective algorithm orchestration<br/>algorithm dispatcher<br/>[hccl_impl.cc]"]
  node_pipeline["Pipeline orchestration<br/>pipeline planner<br/>[hccl_pipeline.cc]"]
  node_operators["Collective operators<br/>operator adapters"]
  node_executor_registry["Executor registry<br/>extension registry"]
  node_collective_executors["Collective executors<br/>operation plans"]
  node_base_communicators["Base communicators<br/>topology planner<br/>[comm_factory.cc]"]
  node_ring_transport["Ring transport requirements<br/>transport planner"]
  node_base_executors["Base algorithm executors<br/>data movement schedules<br/>[all_reduce_ring.cc]"]
end

subgraph group_runtime["Runtime Boundary"]
  node_buffers["CCL buffer manager<br/>workspace resources"]
  node_streams["Offload stream manager<br/>stream resources"]
  node_task_loader["Parallel task loader<br/>async task submission"]
  node_cann{{"CANN platform<br/>external NPU runtime"}}
end

node_public_api -->|"collective calls"| node_communicator
node_build -->|"packages"| node_public_api
node_communicator -->|"creates operations"| node_op_base
node_communicator -->|"initializes ranks"| node_topology
node_communicator -->|"manages transports"| node_transport_manager
node_op_base -->|"dispatches collective"| node_algorithm_impl
node_topology -->|"topology and ranks"| node_algorithm_impl
node_algorithm_impl -->|"pipeline path"| node_pipeline
node_algorithm_impl -->|"selects operator"| node_operators
node_operators -->|"resolves executor"| node_executor_registry
node_executor_registry -->|"instantiates"| node_collective_executors
node_collective_executors -->|"chooses topology"| node_base_communicators
node_base_communicators -->|"calculates requirements"| node_ring_transport
node_collective_executors -->|"composes schedules"| node_base_executors
node_ring_transport -->|"transport requirements"| node_transport_manager
node_base_executors -->|"uses buffers"| node_buffers
node_base_executors -->|"uses streams"| node_streams
node_base_executors -->|"submits tasks"| node_task_loader
node_task_loader -->|"NPU task runtime"| node_cann
node_transport_manager -->|"platform transport"| node_cann
node_buffers -->|"device resources"| node_cann
node_streams -->|"runtime streams"| node_cann

click node_public_api "https://github.com/xwqtju/cann-hccl/blob/main/inc/hccl/hccl.h"
click node_build "https://github.com/xwqtju/cann-hccl/blob/main/CMakeLists.txt"
click node_communicator "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/framework/communicator/impl/hccl_communicator.cc"
click node_op_base "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/framework/op_base/src/op_base.cc"
click node_topology "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/framework/common/src/topo/topoinfo_detect.cc"
click node_transport_manager "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/framework/communicator/impl/resource_manager/transport_manager.cc"
click node_algorithm_impl "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/hccl_impl.cc"
click node_pipeline "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/hccl_pipeline.cc"
click node_operators "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/operator/coll_alg_operator.cc"
click node_executor_registry "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/coll_executor/registry/coll_alg_exec_registry.cc"
click node_collective_executors "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/coll_executor/coll_all_reduce/coll_all_reduce_executor.cc"
click node_base_communicators "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/base/communicator/comm_factory.cc"
click node_ring_transport "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/base/communicator/calc_ring_transport_req.cc"
click node_base_executors "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/base/executor/all_reduce_ring.cc"
click node_buffers "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/resource_manager/ccl_buffer_manager.cc"
click node_streams "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/resource_manager/offload_stream_manager.cc"
click node_task_loader "https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/task/parallel_task_loader.cc"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_public_api,node_build toneBlue
class node_communicator,node_op_base,node_topology,node_transport_manager toneAmber
class node_algorithm_impl,node_pipeline,node_operators,node_executor_registry,node_collective_executors,node_base_communicators,node_ring_transport,node_base_executors toneMint
class node_buffers,node_streams,node_task_loader,node_cann toneRose
```

## Components

### Public & Build

- [HCCL public API](https://github.com/xwqtju/cann-hccl/blob/main/inc/hccl/hccl.h): C++ API · Collective API and public types for consumers. · `inc/hccl/hccl.h`
- [CMake package build](https://github.com/xwqtju/cann-hccl/blob/main/CMakeLists.txt): build entry · Builds the installable HCCL algorithm package. · `CMakeLists.txt`

### Communication Framework

- [HcclCommunicator](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/framework/communicator/impl/hccl_communicator.cc): communicator lifecycle · Owns communication domains and routes calls into algorithm execution. · `src/domain/collective_communication/framework/communicator/impl/hccl_communicator.cc`
- [Operation base](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/framework/op_base/src/op_base.cc): framework operation · `src/domain/collective_communication/framework/op_base/src/op_base.cc`
- [Topology discovery](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/framework/common/src/topo/topoinfo_detect.cc): rank topology · Discovers and exchanges rank-table-derived topology. · `src/domain/collective_communication/framework/common/src/topo/topoinfo_detect.cc`
- [Transport manager](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/framework/communicator/impl/resource_manager/transport_manager.cc): transport lifecycle · Manages framework transport resources. · `src/domain/collective_communication/framework/communicator/impl/resource_manager/transport_manager.cc`

### Algorithm Planning & Execution

- [Collective algorithm orchestration](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/hccl_impl.cc): algorithm dispatcher · Coordinates collective planning and execution. · `src/domain/collective_communication/algorithm/impl/hccl_impl.cc`
- [Pipeline orchestration](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/hccl_pipeline.cc): pipeline planner · Coordinates pipelined collective execution. · `src/domain/collective_communication/algorithm/impl/hccl_pipeline.cc`
- [Collective operators](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/operator/coll_alg_operator.cc): operator adapters · Validate and plan collective operations such as AllReduce and AllToAll. · `src/domain/collective_communication/algorithm/impl/operator/coll_alg_operator.cc`
- [Executor registry](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/coll_executor/registry/coll_alg_exec_registry.cc): extension registry · Selects specialized collective executors. · `src/domain/collective_communication/algorithm/impl/coll_executor/registry/coll_alg_exec_registry.cc`
- [Collective executors](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/coll_executor/coll_all_reduce/coll_all_reduce_executor.cc): operation plans · Concrete collective-family and hardware-specific execution plans. · `src/domain/collective_communication/algorithm/impl/coll_executor/coll_all_reduce/coll_all_reduce_executor.cc`
- [Base communicators](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/base/communicator/comm_factory.cc): topology planner · Builds mesh, ring, star, and point-to-point transport structures. · `src/domain/collective_communication/algorithm/base/communicator/comm_factory.cc`
- [Ring transport requirements](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/base/communicator/calc_ring_transport_req.cc): transport planner · `src/domain/collective_communication/algorithm/base/communicator/calc_ring_transport_req.cc`
- [Base algorithm executors](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/base/executor/all_reduce_ring.cc): data movement schedules · Implements phased send, receive, and reduce schedules. · `src/domain/collective_communication/algorithm/base/executor/all_reduce_ring.cc`

### Runtime Boundary

- [CCL buffer manager](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/resource_manager/ccl_buffer_manager.cc): workspace resources · Manages collective buffers and workspace. · `src/domain/collective_communication/algorithm/impl/resource_manager/ccl_buffer_manager.cc`
- [Offload stream manager](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/resource_manager/offload_stream_manager.cc): stream resources · Manages execution streams. · `src/domain/collective_communication/algorithm/impl/resource_manager/offload_stream_manager.cc`
- [Parallel task loader](https://github.com/xwqtju/cann-hccl/blob/main/src/domain/collective_communication/algorithm/impl/task/parallel_task_loader.cc): async task submission · Loads and submits asynchronous execution tasks. · `src/domain/collective_communication/algorithm/impl/task/parallel_task_loader.cc`
- **CANN platform**: external NPU runtime · Installed toolkit supplies NPU and transport implementation.

## Connections

- HCCL public API → HcclCommunicator: collective calls
- CMake package build → HCCL public API: packages
- HcclCommunicator → Operation base: creates operations
- HcclCommunicator → Topology discovery: initializes ranks
- HcclCommunicator → Transport manager: manages transports
- Operation base → Collective algorithm orchestration: dispatches collective
- Topology discovery → Collective algorithm orchestration: topology and ranks
- Collective algorithm orchestration → Pipeline orchestration: pipeline path
- Collective algorithm orchestration → Collective operators: selects operator
- Collective operators → Executor registry: resolves executor
- Executor registry → Collective executors: instantiates
- Collective executors → Base communicators: chooses topology
- Base communicators → Ring transport requirements: calculates requirements
- Collective executors → Base algorithm executors: composes schedules
- Ring transport requirements → Transport manager: transport requirements
- Base algorithm executors → CCL buffer manager: uses buffers
- Base algorithm executors → Offload stream manager: uses streams
- Base algorithm executors → Parallel task loader: submits tasks
- Parallel task loader → CANN platform: NPU task runtime
- Transport manager → CANN platform: platform transport
- CCL buffer manager → CANN platform: device resources
- Offload stream manager → CANN platform: runtime streams