How nvidia/nccl works
NCCL is a GPU collective-communication library. An application initializes a communicator and submits collective or point-to-point work; NCCL plans that work and executes it across GPUs using device kernels and available local or network transports. The graph also includes topology selection, memory management, remote-memory operations, and optional tuner, profiler, and RAS capabilities. Several implementation links are based on source-index references rather than complete call traces, so those links are dashed; the README supports the documented communication capabilities but not their internal wiring.
Main entry points: enqueue.cc Work submission · comm.h Communicator state · transport.cc Transport selection · core.h NCCL API
Components 20
API and orchestration
- NCCL API
src/include/core.h - Communicator state
src/include/comm.h - Work submission
src/enqueue/enqueue.cc
Planning and scheduling
- Topology search
src/graph/search.cc - Task preparation
src/enqueue/task_prep/task_posttuning.cc - Task scheduling
src/enqueue/task_sched/task_sched.cc
Communication execution
- Collective operations
src/collectives.cc - GPU kernels
src/device/prims_simple.h - Transport selection
src/transport.cc - Network transport
src/transport/net.cc - Proxy progress
src/proxy.cc - Remote memory
src/rma/rma.cc
Resources and observability
- Memory management
src/mem_manager.cc - Core contracts
src/include
Extensions and diagnostics
- Tuning plugins
src/plugin/tuner.cc - Profiler plugins
src/plugin/profiler.cc - Inspector plugin
plugins/profiler/inspector/inspector.cc - RAS monitoring
src/ras/client_support.cc
Other components
- GPU application
- Network fabric
Connections 20
- GPU application → NCCL API: calls
README.md - NCCL API → Communicator state: uses
- Communicator state → Work submission: submits work
- Work submission → Task preparation: prepares tasks
- Work submission → Topology search: uses topology
- Work submission → Task scheduling: schedules work
src/enqueue/enqueue.cc - Task preparation → Tuning plugins: tunes tasks
- Task scheduling → Collective operations: dispatches work
- Collective operations → GPU kernels: executes on GPU
- Work submission → Transport selection: uses transport
src/enqueue/enqueue.cc - Transport selection → Network transport: selects network
- Network transport → Network fabric: transfers data
- Transport selection → Proxy progress: uses progress
- Work submission → Remote memory: dispatches RMA
src/enqueue/enqueue.cc - Communicator state → Memory management: manages memory
src/mem_manager.cc - Work submission → Profiler plugins: reports activity
src/enqueue/enqueue.cc - Profiler plugins → Inspector plugin: loads plugin
src/plugin/profiler.cc - Communicator state → RAS monitoring: reports status
src/ras/client_support.cc - Communicator state → Core contracts: uses definitions
src/include/comm.h - Transport selection → Core contracts: uses interfaces
src/include/transport.h