How infinigence/flashoverlap works
Project type and purpose
• FlashOverlap is an open-source high-performance library/tool for overlapping GEMM computation (via NVIDIA CUTLASS kernels) with NCCL-based communication on GPUs.
• Its primary goal is to provide lightweight building blocks (CUDA kernels + Python bindings) that let you overlap matrix-multiply work and collective communication (e.g., all-reduce) with fine-grain signaling.
File structure
• Top-level:
• CMakeLists.txt + cmake/Modules → build system for native code
• src/ → core CUDA/C++ implementation
• example/ → end-to-end Python demos
• tune/ → scripts to profile hardware & search optimal tilings
• tool/ → generates CUTLASS GEMM instantiations
• test/ → correctness and speed tests
• src/ subdivision:
• 3rdparty/cutlass → CUTLASS submodule
• gemm → wrappers around CUTLASS GEMM kernels
• overlap → signaling infrastructure for compute/comm overlap
• rmsnorm → custom kernel + utilities for RMS normalization
• tiling → dispatchers & tile configurations
• baseline_impl & overlap_impl → C++ classes encapsulating strategies
• nccl_utils → assign NCCL unique IDs
• pybind.cpp → PyTorch custom operator binding
• tune/ scripts use CUTLASS profiler or custom profiling to generate JSON configs stored under configs/.
README for architecture insights
• Two CUDA streams: one for GEMM computation, one for NCCL communication.
• A CUTLASS kernel emits ‘signals’ through a small CUDA counting kernel in the overlap stream; the NCCL all-reduce stalls until enough signals arrive.
• Python layer loads the compiled .so (torch.ops.load_library) and invokes the fused ops in example/test scripts.
Be detailed but not cluttered
• Submodule (gemm, overlap, rmsnorm, tiling) as a compartment inside the GPU layer.
• Two CUDA streams branching from the Native Library:
• Stream 0: CUTLASS GEMM → sends signal events
• Stream 1: Counting kernel → waits → NCCL AllReduce
• Critical data flows (matrix tiles, signals, collective ops).