How axio-project/fuselink works
1. What this repository is
- FuseLink is a NCCL network plugin that replaces/extends NCCL’s transport layer to do efficient GPU communication over multiple RDMA-capable NICs.
- The main architectural boundary is: application / training code → NCCL → FuseLink plugin → RDMA/GPU Direct RDMA → NICs and GPUs.
2. Core runtime component
- The production integration lives in src/plugin.cc, which is the NCCL-facing entry point.
- This is the layer that exposes FuseLink as libnccl-net-fuselink.so and makes it selectable through NCCL_NET_PLUGIN=fuselink.
3. Memory and data movement layer
- src/cumem.cu and src/cumem.h implement CUDA-side memory operations used by the transport.
- This is where GPU memory interaction, peer access, and GPU Direct RDMA-adjacent behavior are centralized, so it sits directly on the data path.
4. FuseLink internal state and transport structures
- src/fuselink.cc and src/fuselink.h hold the plugin’s internal memory/connection structures.
- This is the repository’s main control/data model for tracking how traffic is distributed across NICs and how communication resources are organized.
5. RDMA and networking wrappers
- src/extern/ibvcore.h, ibvwrap.h, and nccl_net.h provide the low-level verbs/NCCL interfaces the plugin builds on.
- src/extern/socket.cc/.h and utils.h are support utilities, likely used for control-plane or fallback communication rather than the fast GPU data path.
- param.h and timer.h are auxiliary support headers borrowed from NCCL.
6. Monitoring subsystem
- src/monitor.cpp, src/monitor.h, and src/monitor_main.cpp implement a separate monitor process/component.
- The README calls this out as monitoring “NIC idleness,” so architecturally this is a side subsystem that observes NIC utilization and likely feeds scheduling/selection decisions in the plugin or related tooling.
7. Build and packaging boundaries
- The repo is split between building NCCL itself and building the FuseLink plugin against that NCCL tree.
- The documented build flow uses make src.build for NCCL and then make fl for the plugin, with NCCL_BUILD_DIR and CUDA_HOME controlling the linkage/runtime environment.
8. Deployment/runtime requirements
- FuseLink depends on RDMA-capable NICs, GPU Direct RDMA, and preferably NVLink for intra-node GPU communication efficiency.
- It also assumes specific software versions: NCCL 2.19.4, libibverbs, CUDA ≥ 11.8, and an NVIDIA driver version tested by the authors.
- The scripts/disable_asc.sh helper is part of the deployment boundary because PCIe ACS must be disabled for fast NIC access.
9. Serving/benchmark application layer
- experiments/serving is a substantial application harness for testing the plugin in model-serving scenarios.
- It includes CMake build files, scheduler logic, model wrappers, online/offline entrypoints, and batch scripts for benchmark execution.
- This directory is where the plugin is exercised under realistic multi-GPU, multi-NIC serving loads.
10. Model-serving data flow
- experiments/serving/main.cc and online_main.cc are the main executables for serving-style experiments.
- model.hh and online_model.hh define the model-side abstraction, while scheduler.h suggests request/task placement or concurrency control around communication-heavy workloads.
- The included SwiftTransformer submodule indicates the repository can integrate with large-model serving code paths.
11. Experimentation and evaluation harness
- experiments/bw_test, experiments/dlrm, and experiments/ep provide benchmark or research-specific testbeds.
- experiments/serving/experiment/common/gen_input.py generates inputs for repeatable experiments, and the .sbatch files indicate the repo is designed to run in scheduled HPC/cluster environments.
- These are not the core runtime, but they define how the communication layer is validated under different workload shapes.
12. External code and submodules
- nccl/ is a checked-in NCCL dependency used for building and plugin integration.
- experiments/ep/Megatron-DeepSpeed and experiments/serving/SwiftTransformer are external model/framework submodules that serve as workload generators or integration targets.
13. How to draw the architecture
- A clear diagram should show three main planes:
- Application plane: training/serving/benchmark apps in experiments/*
- Communication plane: NCCL + FuseLink plugin + monitor
- Hardware plane: GPUs, NICs, RDMA fabric, NVLink/PCIe
- The primary flow is application collective communication requests entering NCCL, being intercepted by FuseLink, then routed over multiple NICs with GPU-aware memory handling and monitoring support.
Main entry points: fuselink.cc FuseLink core · plugin.cc Plugin entry · nccl NCCL tree · cumem.cu GPU memory
Components 19
Application plane
Serving and benchmark workloads that drive NCCL collectives through FuseLink.
- Serving harness · Main model-serving benchmark and execution harness.
experiments/serving - BW test · Bandwidth-focused experiment entrypoint.
experiments/bw_test - DLRM · Recommendation-model experiment target.
experiments/dlrm - EP suite · Experiment area for large-scale model-parallel training.
experiments/ep - SwiftTransformer · External serving/model integration used by the harness.
experiments/serving/SwiftTransformer - Megatron-DeepSpeed · External training stack used in EP experiments.
experiments/ep/Megatron-DeepSpeed
Communication plane
NCCL integration, transport logic, memory movement, and monitoring.
- Plugin entry · NCCL-facing entry point exposed as libnccl-net-fuselink.so.
src/plugin.cc - FuseLink core · Internal connection, resource, and traffic-distribution model.
src/fuselink.cc - GPU memory · CUDA-side memory operations and GPU Direct RDMA-adjacent behavior.
src/cumem.cu - Monitor · Separate process that watches NIC idleness and related signals.
src/monitor_main.cpp
Runtime and external deps
Checked-in NCCL tree and low-level verbs/socket support used by the plugin.
- NCCL tree · Checked-in NCCL dependency and build target for plugin integration.
nccl - Verbs wrappers · Low-level ibverbs and NCCL network interface wrappers.
src/extern/ibvwrap.h - IBV core · Core verbs helpers and RDMA plumbing used by the plugin.
src/extern/ibvcore.h - Socket support · Auxiliary socket utilities for setup or fallback communication.
src/extern/socket.cc - ACS disable · Helper for disabling PCIe ACS to improve device access.
scripts/disable_asc.sh
Hardware plane
GPU, NIC, and interconnect resources targeted by the plugin.
- RDMA fabric · Multi-NIC RDMA path used for transport traffic.
- GPUs · GPU devices providing collective communication endpoints and memory.
- NICs · Multiple network interfaces used for load distribution.
- NVLink/PCIe · Local GPU interconnect and PCIe access constraints.
Connections 21
- Serving harness → NCCL tree: uses NCCL · Serving workloads rely on NCCL collectives.
- BW test → NCCL tree: uses NCCL · Benchmark harness drives the communication stack.
- DLRM → NCCL tree: uses NCCL · Training benchmark exercises NCCL transports.
- EP suite → NCCL tree: uses NCCL · Model-parallel experiments depend on NCCL.
- Serving harness → SwiftTransformer: integrates · Serving harness wraps the transformer workload.
- EP suite → Megatron-DeepSpeed: integrates · EP experiments use Megatron-DeepSpeed as workload source.
- NCCL tree → Plugin entry: loads plugin · NCCL resolves the FuseLink network plugin.
- Plugin entry → FuseLink core: delegates · Entry point hands transport setup and operations to core state.
- FuseLink core → GPU memory: uses · Core transport relies on CUDA-side memory movement.
- FuseLink core → Verbs wrappers: uses · Core transport builds on RDMA verbs abstractions.
- Verbs wrappers → IBV core: wraps · Wrapper layer sits on lower-level verbs helpers.
- Plugin entry → Socket support: control path · Plugin may use socket utilities for auxiliary communication.
- Monitor → FuseLink core: observes · Monitor tracks NIC idleness relevant to plugin decisions.
- Monitor → NICs: measures · Side process watches NIC utilization.
- FuseLink core → NICs: spreads traffic · Transport distributes communication across multiple NICs.
- FuseLink core → RDMA fabric: sends over · Primary data path rides the RDMA fabric.
- GPU memory → GPUs: accesses · Memory layer operates on GPU-resident buffers.
- GPU memory → NVLink/PCIe: depends on · Efficient local transfers depend on GPU interconnect and PCIe behavior.
- ACS disable → NVLink/PCIe: configures · Deployment helper disables ACS to improve PCIe access paths.
- Plugin entry → GPUs: serves · Plugin exposes GPU-aware transport to the application stack.
- Plugin entry → NICs: serves · Plugin routes communication through RDMA-capable NICs.