# infinigence/flashoverlap architecture

> Architecture diagram of the GitHub repository infinigence/flashoverlap, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/infinigence/flashoverlap
- Repository: https://github.com/infinigence/flashoverlap

## Overview

##### Project type and purpose
- FlashOverlap is an open-source high-performance library/tool for overlapping GEMM computation (via NVIDIA CUTLASS kernels) with NCCL-based communication on GPUs.
- Its primary goal is to provide lightweight building blocks (CUDA kernels + Python bindings) that let you overlap matrix-multiply work and collective communication (e.g., all-reduce) with fine-grain signaling.

##### File structure
- Top-level:
- CMakeLists.txt + cmake/Modules → build system for native code
- src/ → core CUDA/C++ implementation
- example/ → end-to-end Python demos
- tune/ → scripts to profile hardware & search optimal tilings
- tool/ → generates CUTLASS GEMM instantiations
- test/ → correctness and speed tests
- src/ subdivision:
- 3rdparty/cutlass → CUTLASS submodule
- gemm → wrappers around CUTLASS GEMM kernels
- overlap → signaling infrastructure for compute/comm overlap
- rmsnorm → custom kernel + utilities for RMS normalization
- tiling → dispatchers & tile configurations
- baseline_impl & overlap_impl → C++ classes encapsulating strategies
- nccl_utils → assign NCCL unique IDs
- pybind.cpp → PyTorch custom operator binding
- tune/ scripts use CUTLASS profiler or custom profiling to generate JSON configs stored under configs/.

##### README for architecture insights
- Two CUDA streams: one for GEMM computation, one for NCCL communication.
- A CUTLASS kernel emits ‘signals’ through a small CUDA counting kernel in the overlap stream; the NCCL all-reduce stalls until enough signals arrive.
- Python layer loads the compiled `.so` (torch.ops.load_library) and invokes the fused ops in example/test scripts.

##### Be detailed but not cluttered
- Submodule (gemm, overlap, rmsnorm, tiling) as a compartment inside the GPU layer.
- Two CUDA streams branching from the Native Library:
- Stream 0: CUTLASS GEMM → sends signal events
- Stream 1: Counting kernel → waits → NCCL AllReduce
- Critical data flows (matrix tiles, signals, collective ops).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Build System Layer
    subgraph "Build System"
        direction TB
        BS1["CMakeLists.txt"] 
        BS2["cmake/Modules/FindNCCL.cmake"] 
        BS3["src/CMakeLists.txt"] 
    end
    class BS1,BS2,BS3 build

    %% Native/GPU Library Layer
    subgraph "Native/GPU Library"
        direction TB
        %% Core Modules
        subgraph "GEMM Wrappers"
            direction TB
            GEMM_CU["gemm.cu"] 
            GEMM_H["gemm.h"] 
            GEMM_INC["gemm_instances.inc"] 
        end
        subgraph "Overlap Logic"
            direction TB
            SIG_CU["gemm_signal.cu"] 
            SIG_H["gemm_signal.h"] 
            SIG_WITH_H["gemm_with_signal.h"] 
            OV_IMPL_CU["overlap_impl.cu"] 
            OV_IMPL_H["overlap_impl.h"] 
        end
        subgraph "RMSNorm Kernels"
            direction TB
            RMS_CU["rmsnorm.cu"] 
            RMS_CUH["rmsnorm.cuh"] 
            RMS_H["rmsnorm.h"] 
            RMS_UTILS["utils.h"] 
        end
        subgraph "NCCL Utilities"
            direction TB
            NCCL_CU["nccl_utils.cu"] 
            NCCL_H["nccl_utils.h"] 
        end
        subgraph "Baseline Implementation"
            direction TB
            BASE_CU["baseline_impl.cu"] 
            BASE_H["baseline_impl.h"] 
        end
        subgraph "Tiling & Dispatch"
            direction TB
            DISP_H["gemm_dispatcher.h"] 
            TILE_CUH["gemm_tiling.cuh"] 
            SIG_TILE["signal_tiling.cuh"] 
            WAIT_CUH["wait.cuh"] 
        end
        %% GPU Streams
        subgraph "CUDA Streams"
            direction TB
            Compute_Stream["Compute Stream"]
            Overlap_Stream["Overlap Stream"]
        end
    end
    class GEMM_CU,GEMM_H,GEMM_INC,SIG_CU,SIG_H,SIG_WITH_H,OV_IMPL_CU,OV_IMPL_H,RMS_CU,RMS_CUH,RMS_H,RMS_UTILS,NCCL_CU,NCCL_H,BASE_CU,BASE_H,DISP_H,TILE_CUH,SIG_TILE,WAIT_CUH native
    class Compute_Stream,Overlap_Stream native

    %% Python Interface Layer
    subgraph "Python Interface & Scripts"
        direction TB
        PYBIND["pybind.cpp"] 
        subgraph "Examples & Tests"
            direction TB
            EX1["example/RMSNorm.py"]
            EX2["example/RowParallelLinear.py"]
            EX3["example/correctness.py"]
            EX4["example/utils.py"]
            TEST["test/test.py"]
        end
    end
    class PYBIND,EX1,EX2,EX3,EX4,TEST python

    %% Tooling & Tuning Layer
    subgraph "Tooling & Tuning"
        direction TB
        GEN_INST["tool/generate_instances.py"] 
        subgraph "Tuning Scripts"
            direction TB
            TUNE1["tune/bandwidth.py"]
            TUNE2["tune/gen_config.py"]
            TUNE3["tune/profile_config.py"]
            TUNE4["tune/search.py"]
        end
        JSON_CFG["configs/*.json"]
    end
    class GEN_INST,TUNE1,TUNE2,TUNE3,TUNE4,JSON_CFG tool

    %% Dependencies & Data Flows
    BS1 -->|compiles| GEMM_CU
    BS2 -->|finds NCCL| NCCL_CU
    BS3 -->|includes| DISP_H

    GEMM_CU -->|wrapper| Compute_Stream
    Compute_Stream -->|emits signals| Overlap_Stream
    SIG_CU -->|count kernel| Overlap_Stream
    Overlap_Stream -->|triggers| NCCL_CU

    PYBIND -->|loads| GEMM_CU
    PYBIND -->|binds| SIG_CU
    PYBIND -->|binds| RMS_CU

    EX1 -->|calls| PYBIND
    EX2 -->|calls| PYBIND
    EX3 -->|calls| PYBIND
    EX4 -->|imports| PYBIND
    TEST -->|runs| PYBIND

    GEN_INST -->|generates| GEMM_INC
    TUNE1 -->|profiles| JSON_CFG
    TUNE2 -->|writes| JSON_CFG
    TUNE3 -->|profiles| JSON_CFG
    TUNE4 -->|searches| JSON_CFG
    JSON_CFG -->|configures| DISP_H
    JSON_CFG -->|configures| SIG_TILE

    %% Click Events
    click BS1 "https://github.com/infinigence/flashoverlap/blob/main/CMakeLists.txt"
    click BS2 "https://github.com/infinigence/flashoverlap/blob/main/cmake/Modules/FindNCCL.cmake"
    click BS3 "https://github.com/infinigence/flashoverlap/blob/main/src/CMakeLists.txt"

    click GEMM_CU "https://github.com/infinigence/flashoverlap/blob/main/src/gemm/gemm.cu"
    click GEMM_H "https://github.com/infinigence/flashoverlap/blob/main/src/gemm/gemm.h"
    click GEMM_INC "https://github.com/infinigence/flashoverlap/blob/main/src/inc/gemm_instances.inc"

    click SIG_CU "https://github.com/infinigence/flashoverlap/blob/main/src/overlap/gemm_signal.cu"
    click SIG_H "https://github.com/infinigence/flashoverlap/blob/main/src/overlap/gemm_signal.h"
    click SIG_WITH_H "https://github.com/infinigence/flashoverlap/blob/main/src/overlap/gemm_with_signal.h"
    click OV_IMPL_CU "https://github.com/infinigence/flashoverlap/blob/main/src/overlap_impl.cu"
    click OV_IMPL_H "https://github.com/infinigence/flashoverlap/blob/main/src/overlap_impl.h"

    click RMS_CU "https://github.com/infinigence/flashoverlap/blob/main/src/rmsnorm/rmsnorm.cu"
    click RMS_CUH "https://github.com/infinigence/flashoverlap/blob/main/src/rmsnorm/rmsnorm.cuh"
    click RMS_H "https://github.com/infinigence/flashoverlap/blob/main/src/rmsnorm/rmsnorm.h"
    click RMS_UTILS "https://github.com/infinigence/flashoverlap/blob/main/src/rmsnorm/utils.h"

    click NCCL_CU "https://github.com/infinigence/flashoverlap/blob/main/src/nccl_utils.cu"
    click NCCL_H "https://github.com/infinigence/flashoverlap/blob/main/src/nccl_utils.h"

    click BASE_CU "https://github.com/infinigence/flashoverlap/blob/main/src/baseline_impl.cu"
    click BASE_H "https://github.com/infinigence/flashoverlap/blob/main/src/baseline_impl.h"

    click DISP_H "https://github.com/infinigence/flashoverlap/blob/main/src/tiling/gemm_dispatcher.h"
    click TILE_CUH "https://github.com/infinigence/flashoverlap/blob/main/src/tiling/gemm_tiling.cuh"
    click SIG_TILE "https://github.com/infinigence/flashoverlap/blob/main/src/tiling/signal_tiling.cuh"
    click WAIT_CUH "https://github.com/infinigence/flashoverlap/blob/main/src/tiling/wait.cuh"

    click PYBIND "https://github.com/infinigence/flashoverlap/blob/main/src/pybind.cpp"

    click GEN_INST "https://github.com/infinigence/flashoverlap/blob/main/tool/generate_instances.py"

    click TUNE1 "https://github.com/infinigence/flashoverlap/blob/main/tune/bandwidth.py"
    click TUNE2 "https://github.com/infinigence/flashoverlap/blob/main/tune/gen_config.py"
    click TUNE3 "https://github.com/infinigence/flashoverlap/blob/main/tune/profile_config.py"
    click TUNE4 "https://github.com/infinigence/flashoverlap/blob/main/tune/search.py"

    click EX1 "https://github.com/infinigence/flashoverlap/blob/main/example/RMSNorm.py"
    click EX2 "https://github.com/infinigence/flashoverlap/blob/main/example/RowParallelLinear.py"
    click EX3 "https://github.com/infinigence/flashoverlap/blob/main/example/correctness.py"
    click EX4 "https://github.com/infinigence/flashoverlap/blob/main/example/utils.py"
    click TEST "https://github.com/infinigence/flashoverlap/blob/main/test/test.py"

    %% Styles
    classDef build fill:#cccccc,stroke:#333,stroke-width:1px
    classDef native fill:#ffcc99,stroke:#333,stroke-width:1px
    classDef python fill:#99ccff,stroke:#333,stroke-width:1px
    classDef tool fill:#99ff99,stroke:#333,stroke-width:1px
```
