# fw-ai/deepep architecture

> Architecture diagram of the GitHub repository fw-ai/deepep, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fw-ai/deepep
- Repository: https://github.com/fw-ai/deepep

## Overview

##### Project type and purpose
- DeepEP is an open-source GPU communication library for Mixture-of-Experts (MoE) and expert parallelism (EP).
- It provides two families of high-throughput, low-latency “dispatch” and “combine” kernels for intra-GPU (NVLink) and inter-GPU (RDMA) communication, plus a hook-based overlap mechanism.

##### File structure
- Top-level:
- csrc/ : C++ host code, CUDA kernels, CMake build files
- deep_ep/ : Python wrapper module (Buffer, EventOverlap, utils)
- third-party/ : patched NVSHMEM dependency
- setup.py : build/install script for Python + C++ extension
- tests/ : Python tests for intra-node, inter-node, low-latency
- Patterns:
- Host library (C++) + Python front end
- Kernel packaging under csrc/kernels (.cu/.cuh)
- Dependency on external NVSHMEM library and RDMA network
- Two modes: “normal” vs “low-latency” (hook interface, pure RDMA)

##### README for additional insights
- Modes of operation:
- Normal kernels use NVLink for intra-node and RDMA forwarding for inter-node
- Low-latency kernels lean on pure RDMA + hook-based overlap
- External dependencies:
- PyTorch (2.1+), CUDA (12.3+), Hopper GPUs, NVSHMEM, RDMA networking
- Runtime flow: Python → C++ extension → CUDA kernels → network transport
- Overlap: CUDA events and hooks to overlap compute + communication

## Diagram (Mermaid)

```mermaid
graph TB
    %% Application
    App["User Application (PyTorch Model)"]:::external
    CPU["CPU Host"]:::hardware
    GPU["GPU Devices"]:::hardware

    %% Python Layer
    subgraph "Python API Layer"
        Buf["deep_ep Buffer"]:::pythonLayer
        Utils["deep_ep Utils"]:::pythonLayer
        Init["deep_ep __init__"]:::pythonLayer
    end

    %% C++ Extension
    subgraph "C++ Extension"
        ExtCpp["deep_ep_cpp.so"]:::cppHost
        DeepEpCpp["deep_ep.cpp"]:::cppHost
        DeepEpHpp["deep_ep.hpp"]:::cppHost
        CMakeExt["CMakeLists.txt"]:::cppHost
        SetupPy["setup.py"]:::cppHost
    end

    %% C++ Host Utilities
    subgraph "C++ Host Utilities"
        ConfigHpp["config.hpp"]:::cppHost
        EventHpp["event.hpp"]:::cppHost
    end

    %% CUDA Kernels
    subgraph "CUDA Kernel Modules"
        Runtime["runtime.cu"]:::kernel
        IntraNode["intranode.cu"]:::kernel
        InterNode["internode.cu"]:::kernel
        InterNodeLL["internode_ll.cu"]:::kernel
        APIcuh["api.cuh"]:::kernel
        BufferCuh["buffer.cuh"]:::kernel
        ConfigsCuh["configs.cuh"]:::kernel
        ExceptionCuh["exception.cuh"]:::kernel
        IBGdaCuh["ibgda_device.cuh"]:::kernel
        LaunchCuh["launch.cuh"]:::kernel
        UtilsCuh["utils.cuh"]:::kernel
        KernelsCMake["CMakeLists.txt"]:::kernel
    end

    %% Third-Party
    subgraph "Third-Party NVSHMEM"
        NVShmemPatch["nvshmem.patch"]:::external
        NVShmemReadme["README.md"]:::external
    end

    %% Tests
    subgraph "Test Suites"
        TestIntra["test_intranode.py"]:::test
        TestInter["test_internode.py"]:::test
        TestLL["test_low_latency.py"]:::test
        TestUtils["utils.py"]:::test
    end

    %% Documentation
    README["README.md"]:::external
    Figures["figures/"]:::external
    License["LICENSE"]:::external
    Gitignore[".gitignore"]:::external

    %% External Networks/Libraries
    NVLink["NVLink Network"]:::external
    RDMA["RDMA Network"]:::external
    NVSHMEM["NVSHMEM Library"]:::external

    %% Relationships
    App -->|"calls Buffer.dispatch/combine"| Buf
    App -->|"imports Utils"| Utils
    App -->|"imports __init__"| Init
    Buf -->|"Python C API"| ExtCpp
    Utils -->|"Python C API"| ExtCpp
    ExtCpp -->|"includes DeepEP C++ code"| DeepEpCpp
    ExtCpp -->|"includes headers"| DeepEpHpp
    ExtCpp -->|"build via"| CMakeExt
    ExtCpp -->|"setup via"| SetupPy
    ExtCpp -->|"uses host utils"| ConfigHpp
    ExtCpp -->|"uses event sync"| EventHpp
    ExtCpp -->|"launches kernels"| Runtime
    Runtime -->|"calls"| IntraNode
    Runtime -->|"calls"| InterNode
    Runtime -->|"calls low-latency"| InterNodeLL
    IntraNode -->|"uses API defs"| APIcuh
    IntraNode -->|"uses buffer mgmt"| BufferCuh
    IntraNode -->|"uses configs"| ConfigsCuh
    IntraNode -->|"handles errors"| ExceptionCuh
    IntraNode -->|"interfaces IB device"| IBGdaCuh
    IntraNode -->|"uses launch helper"| LaunchCuh
    IntraNode -->|"utils"| UtilsCuh
    InterNode -->|"similar deps"| APIcuh
    InterNodeLL -->|"similar deps"| APIcuh
    Runtime -->|"build via"| KernelsCMake
    IntraNode -->|"over NVLink"| NVLink
    IntraNode -->|"via NVSHMEM"| NVSHMEM
    InterNode -->|"over RDMA"| RDMA
    InterNode -->|"via NVSHMEM"| NVSHMEM
    InterNodeLL -->|"over RDMA hooks"| RDMA
    CPU -->|"runs"| ExtCpp
    GPU -->|"executes"| Runtime
    GPU -->|"executes"| IntraNode
    GPU -->|"executes"| InterNode
    GPU -->|"executes"| InterNodeLL
    NVShmemPatch -->|"patches"| NVSHMEM
    NVShmemReadme -->|"docs"| NVSHMEM
    TestIntra -->|"tests"| IntraNode
    TestInter -->|"tests"| InterNode
    TestLL -->|"tests"| InterNodeLL
    TestUtils -->|"helpers"| UtilsCuh
    README -->|"docs project"| ExtCpp
    Figures -->|"architecture/perf"| README
    License -->|"project license"| README
    Gitignore -->|"VCS ignore"| README

    %% Click Events
    click Buf "https://github.com/fw-ai/deepep/blob/main/deep_ep/buffer.py"
    click Utils "https://github.com/fw-ai/deepep/blob/main/deep_ep/utils.py"
    click Init "https://github.com/fw-ai/deepep/blob/main/deep_ep/__init__.py"
    click DeepEpCpp "https://github.com/fw-ai/deepep/blob/main/csrc/deep_ep.cpp"
    click DeepEpHpp "https://github.com/fw-ai/deepep/blob/main/csrc/deep_ep.hpp"
    click CMakeExt "https://github.com/fw-ai/deepep/blob/main/csrc/CMakeLists.txt"
    click SetupPy "https://github.com/fw-ai/deepep/blob/main/setup.py"
    click ConfigHpp "https://github.com/fw-ai/deepep/blob/main/csrc/config.hpp"
    click EventHpp "https://github.com/fw-ai/deepep/blob/main/csrc/event.hpp"
    click IntraNode "https://github.com/fw-ai/deepep/blob/main/csrc/kernels/intranode.cu"
    click InterNode "https://github.com/fw-ai/deepep/blob/main/csrc/kernels/internode.cu"
    click InterNodeLL "https://github.com/fw-ai/deepep/blob/main/csrc/kernels/internode_ll.cu"
    click Runtime "https://github.com/fw-ai/deepep/blob/main/csrc/kernels/runtime.cu"
    click APIcuh "https://github.com/fw-ai/deepep/blob/main/csrc/kernels/api.cuh"
    click BufferCuh "https://github.com/fw-ai/deepep/blob/main/csrc/kernels/buffer.cuh"
    click ConfigsCuh "https://github.com/fw-ai/deepep/blob/main/csrc/kernels/configs.cuh"
    click ExceptionCuh "https://github.com/fw-ai/deepep/blob/main/csrc/kernels/exception.cuh"
    click IBGdaCuh "https://github.com/fw-ai/deepep/blob/main/csrc/kernels/ibgda_device.cuh"
    click LaunchCuh "https://github.com/fw-ai/deepep/blob/main/csrc/kernels/launch.cuh"
    click UtilsCuh "https://github.com/fw-ai/deepep/blob/main/csrc/kernels/utils.cuh"
    click KernelsCMake "https://github.com/fw-ai/deepep/blob/main/csrc/kernels/CMakeLists.txt"
    click NVShmemPatch "https://github.com/fw-ai/deepep/blob/main/third-party/nvshmem.patch"
    click NVShmemReadme "https://github.com/fw-ai/deepep/blob/main/third-party/README.md"
    click TestIntra "https://github.com/fw-ai/deepep/blob/main/tests/test_intranode.py"
    click TestInter "https://github.com/fw-ai/deepep/blob/main/tests/test_internode.py"
    click TestLL "https://github.com/fw-ai/deepep/blob/main/tests/test_low_latency.py"
    click TestUtils "https://github.com/fw-ai/deepep/blob/main/tests/utils.py"
    click README "https://github.com/fw-ai/deepep/blob/main/README.md"
    click Figures "https://github.com/fw-ai/deepep/tree/main/figures/"
    click License "https://github.com/fw-ai/deepep/tree/main/LICENSE"
    click Gitignore "https://github.com/fw-ai/deepep/blob/main/.gitignore"

    %% Styles
    classDef pythonLayer fill:#cce5ff,stroke:#1f77b4
    classDef cppHost fill:#d4edda,stroke:#2ca02c
    classDef kernel fill:#ffe5cc,stroke:#ff7f0e
    classDef external fill:#e6ccff,stroke:#9467bd
    classDef hardware fill:#f0e68c,stroke:#8c564b
    classDef test fill:#f0f0f0,stroke:#7f7f7f
```
