How fw-ai/deepep works
Project type and purpose
• DeepEP is an open-source GPU communication library for Mixture-of-Experts (MoE) and expert parallelism (EP).
• It provides two families of high-throughput, low-latency “dispatch” and “combine” kernels for intra-GPU (NVLink) and inter-GPU (RDMA) communication, plus a hook-based overlap mechanism.
File structure
• Top-level:
• csrc/ : C++ host code, CUDA kernels, CMake build files
• deep_ep/ : Python wrapper module (Buffer, EventOverlap, utils)
• third-party/ : patched NVSHMEM dependency
• setup.py : build/install script for Python + C++ extension
• tests/ : Python tests for intra-node, inter-node, low-latency
• Patterns:
• Host library (C++) + Python front end
• Kernel packaging under csrc/kernels (.cu/.cuh)
• Dependency on external NVSHMEM library and RDMA network
• Two modes: “normal” vs “low-latency” (hook interface, pure RDMA)
README for additional insights
• Modes of operation:
• Normal kernels use NVLink for intra-node and RDMA forwarding for inter-node
• Low-latency kernels lean on pure RDMA + hook-based overlap
• External dependencies:
• PyTorch (2.1+), CUDA (12.3+), Hopper GPUs, NVSHMEM, RDMA networking
• Runtime flow: Python → C++ extension → CUDA kernels → network transport
• Overlap: CUDA events and hooks to overlap compute + communication