xwqtju/cann-hccl

Purpose and Entry Points

cann-hccl is Huawei’s C++ collective-communication implementation for Ascend NPUs, covering single- and multi-node data/model-parallel operations. This repository supplies the communication framework and algorithm layers; the underlying communication platform is provided by the installed CANN toolkit.

Public consumer-facing API/types are under inc/hccl/hccl.h and inc/hccl/hccl_types.h. The build/package entry point is build.sh, with the root build graph in CMakeLists.txt.

Layered Execution Path

A collective call is routed through framework communicator management, algorithm/operator selection, resource preparation, and task execution:

1. Framework initializes the communication domain and communicator.

2. Topology/rank information and operator parameters drive algorithm selection.

3. Operators obtain a specialized collective executor from registries.

4. Executors compose topology-specific base algorithms and transport requirements.

5. Tasks are submitted using streams, buffers, queues/notifies, and platform-backed transport resources.

Key orchestration paths: src/domain/collective_communication/framework/communicator/impl/hccl_communicator.cc, src/domain/collective_communication/algorithm/impl/hccl_impl.cc, and src/domain/collective_communication/algorithm/impl/hccl_pipeline.cc.

Framework: Communicators and Topology

The framework owns communication-domain lifecycle and bridges application calls to algorithm execution. HcclCommunicator and transport management form the principal boundary to the CANN communication platform. Relevant code: src/domain/collective_communication/framework/communicator/impl/hccl_communicator.cc, src/domain/collective_communication/framework/communicator/impl/resource_manager/transport_manager.cc, and src/domain/collective_communication/framework/op_base/src/op_base.cc.

Topology discovery, rank-table parsing, whitelist handling, and topology exchange are centralized separately. These inputs determine rank relationships and topology-aware planning: src/domain/collective_communication/framework/common/src/topo/topoinfo_detect.cc, src/domain/collective_communication/framework/common/src/topo/topoinfo_ranktableParser.cc, and src/domain/collective_communication/framework/common/src/topo/topoinfo_exchange_dispatcher.cc.

Operator Selection and Algorithm Dispatch

Each collective has an operator adapter responsible for validation/planning integration, including AllReduce, AllGather, AllToAll, Broadcast, Reduce, ReduceScatter, Scatter, and point-to-point operations. Operators and executors are selected through registries, creating the main extensibility boundary for custom operations/algorithms.

Grounding paths: src/domain/collective_communication/algorithm/impl/operator/coll_alg_operator.cc, src/domain/collective_communication/algorithm/impl/operator/all_reduce_operator.cc, and src/domain/collective_communication/algorithm/impl/coll_executor/registry/coll_alg_exec_registry.cc.

Collective Executors

Operation-specific executors implement concrete plans for collective families and hardware/count variants. For example, AllReduce has ring, mesh, one-shot, pipeline, RDMA/AIV, and Ascend 310P-specific executors; comparable families exist for AllGather, Broadcast, ReduceScatter, AllToAll, Reduce, and Scatter.

Representative paths: src/domain/collective_communication/algorithm/impl/coll_executor/coll_all_reduce/coll_all_reduce_executor.cc, src/domain/collective_communication/algorithm/impl/coll_executor/coll_all_to_all/coll_all_to_all_v_staged_executor.cc, and src/domain/collective_communication/algorithm/impl/coll_executor/coll_reduce_scatter/coll_reduce_scatter_ring_executor.cc.

Base Topologies and Data Movement

Reusable base communicators calculate transport requirements and encode Mesh, Ring, point-to-point, Star, and recursive halving-doubling communication structures. Base executors implement the actual phased send/receive/reduce schedules, including direct, atomic, concurrent, and staged variants.

Key paths: src/domain/collective_communication/algorithm/base/communicator/comm_factory.cc, src/domain/collective_communication/algorithm/base/communicator/calc_ring_transport_req.cc, and src/domain/collective_communication/algorithm/base/executor/all_reduce_ring.cc.

Resource and Runtime Boundary

Algorithm execution depends on managed CCL buffers/workspace, sockets, streams, queue notifications, and asynchronous task loading. These are the material runtime boundary between planner/executor code and CANN’s NPU/transport services; this repository does not contain the platform implementation itself.

Relevant paths: src/domain/collective_communication/algorithm/impl/resource_manager/ccl_buffer_manager.cc, src/domain/collective_communication/algorithm/impl/resource_manager/offload_stream_manager.cc, and src/domain/collective_communication/algorithm/impl/task/parallel_task_loader.cc.

Build and External Dependencies

The project builds with CMake/GCC/Python on Linux and produces an installable CANN HCCL algorithm package. It requires a version-compatible CANN development toolkit and external nlohmann/json headers; runtime validation is performed by the separately supplied HCCL Test/MPI tooling. Build integration is defined in CMakeLists.txt, cmake/config.cmake, and cmake/intf_pub_linux.cmake.