# bytedance/flux architecture

> Architecture diagram of the GitHub repository bytedance/flux, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bytedance/flux
- Repository: https://github.com/bytedance/flux

## Overview

##### Project Type and Purpose
- Flux is a high-performance, fine-grained GPU kernel library with Python bindings for PyTorch.
- It fuses computation and communication (dense GEMM, MoE kernels, all-gather, reduce-scatter, etc.) to maximize GPU utilization and overlap.

##### Main Components
a. Third-Party Dependencies
- 3rdparty/CUTLASS (tensor-core GEMM templates)
- 3rdparty/NCCL (multi-GPU collectives)
- NVSHMEM (optional, for MoE kernels)
b. Core C++/CUDA Kernel Modules (src/…)
- gemm_only, ag_gemm, gemm_rs, moe_ag_scatter, moe_gather_rs, a2a_transpose_gemm, coll, comm_none, inplace_cast, quantization
- Each sub-folder contains:
– CUDA kernel implementations (.cu/.cuh)
– Host-side glue (ths_op)
– Tuning configurations for different GPU architectures
c. Code Generator (src/generator/)
- Reads high-level descriptions and emits specialized C++ kernels for each submodule
d. Public API Headers (include/flux/)
- op_registry, runtime_config, utils, dispatch policies
- These define the compile-time and run-time interfaces
e. Python Bindings
- C++ extension (src/pybind/ using pybind11)
- Python packages: python/flux (user API), python/flux_triton (Triton backend)
f. Build & Packaging
- CMake (CMakeLists.txt, cmake/)
- build.sh, Dockerfile, install_deps.sh, setup.py, pyproject.toml
- Produces: shared libraries, Python wheel
g. User Layer
- Examples (examples/)
- Tests (test/python/, test/unit/, test/tools/)
- Documentation (docs/, README.md)

##### Component Relationships & Data Flow
- Code Generator → emits C++ kernel source → compiled by Build System
- Build System (CMake/build.sh) → compiles 3rdparty + kernel modules → links into shared libs + Python extension
- Core C++ modules register kernels via op_registry
- PyBind extension loads C++ modules → exposes Flux operations to Python
- Python flux API calls into extension → launches fused CUDA kernels
- Kernels coordinate via NCCL/NVSHMEM for inter-GPU communication and CUTLASS for compute
- User scripts/tests/examples invoke Python API → drive GPU workloads

##### Architectural Patterns & Design Principles
- Modular submodule structure: one folder per kernel family, consistent layout
- Plugin-style op registry for decoupled implementation and dispatch
- Build-time code generation for specialized kernels
- Multi-language layering: C++/CUDA core + Python wrapper
- Separation of concerns: compute, communication, configuration, binding

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Hardware & Drivers"
        GPU["GPU"]:::hardware
        CUDA["CUDA Runtime"]:::hardware
        NVSHMEM_HW["NVSHMEM Driver"]:::hardware
    end

    subgraph "3rd-Party Libraries"
        CUTLASS[(CUTLASS)]:::thirdparty
        NCCL[(NCCL)]:::thirdparty
        NVSHMEM_LIB[(NVSHMEM)]:::thirdparty
    end

    subgraph "Core Kernel Modules"
        gemm_only["comm_none (gemm_only)"]:::core
        ag_gemm["ag_gemm"]:::core
        gemm_rs["gemm_rs"]:::core
        moe_ag_scatter["moe_ag_scatter"]:::core
        moe_gather_rs["moe_gather_rs"]:::core
        a2a_transpose_gemm["a2a_transpose_gemm"]:::core
        coll["coll"]:::core
        comm_none["comm_none"]:::core
        inplace_cast["inplace_cast"]:::core
        quantization["quantization"]:::core
    end

    Generator["Code Generator"]:::core

    subgraph "Build & Packaging"
        CMakeLists["CMakeLists.txt"]:::build
        cmake["cmake/"]:::build
        build_sh["build.sh"]:::build
        install_deps["install_deps.sh"]:::build
        Dockerfile["Dockerfile"]:::build
        setup_py["setup.py"]:::build
        pyproject["pyproject.toml"]:::build
        setup_cfg["setup.cfg"]:::build
        manifest["MANIFEST.in"]:::build
    end

    pybind["PyBind Extension"]:::binding

    subgraph "Python API Layer"
        python_flux["python/flux"]:::python
        python_triton["python/flux_triton"]:::python
    end

    subgraph "User Layer"
        examples["examples/"]:::python
        test_py["test/python/"]:::python
        test_unit["test/unit/"]:::python
        test_tools["test/tools/"]:::python
        docs["docs/ & README"]:::python
    end

    %% Data/Control Flow
    GPU --> CUTLASS
    GPU --> NCCL
    GPU --> NVSHMEM_LIB

    CUTLASS -->|compute templates| gemm_only
    CUTLASS --> ag_gemm
    CUTLASS --> gemm_rs
    CUTLASS --> a2a_transpose_gemm
    NCCL -->|collectives| coll
    NCCL --> gemm_rs
    NCCL --> moe_ag_scatter
    NCCL --> moe_gather_rs
    NVSHMEM_LIB -->|optional comm| moe_ag_scatter
    NVSHMEM_LIB --> moe_gather_rs

    Generator -->|emits kernels| gemm_only
    Generator --> ag_gemm
    Generator --> gemm_rs
    Generator --> moe_ag_scatter
    Generator --> moe_gather_rs
    Generator --> a2a_transpose_gemm
    Generator --> coll
    Generator --> comm_none
    Generator --> inplace_cast
    Generator --> quantization

    CMakeLists -.-> CUTLASS
    CMakeLists -.-> NCCL
    CMakeLists -.-> NVSHMEM_LIB
    CMakeLists -->|compiles| gemm_only
    CMakeLists --> ag_gemm
    CMakeLists --> gemm_rs
    CMakeLists --> moe_ag_scatter
    CMakeLists --> moe_gather_rs
    CMakeLists --> a2a_transpose_gemm
    CMakeLists --> coll
    CMakeLists --> comm_none
    CMakeLists --> inplace_cast
    CMakeLists --> quantization
    CMakeLists --> pybind

    pybind -->|pybind11| python_flux
    pybind -->|pybind11| python_triton

    python_flux -->|user API| examples
    python_flux --> test_py
    python_flux --> test_unit
    python_flux --> test_tools
    python_flux --> docs
    python_triton --> examples

    %% Click Events
    click CUTLASS "https://github.com/bytedance/flux/tree/main/3rdparty/cutlass"
    click NCCL "https://github.com/bytedance/flux/tree/main/3rdparty/nccl"
    click gemm_only "https://github.com/bytedance/flux/tree/main/src/comm_none"
    click ag_gemm "https://github.com/bytedance/flux/tree/main/src/ag_gemm"
    click gemm_rs "https://github.com/bytedance/flux/tree/main/src/gemm_rs"
    click moe_ag_scatter "https://github.com/bytedance/flux/tree/main/src/moe_ag_scatter"
    click moe_gather_rs "https://github.com/bytedance/flux/tree/main/src/moe_gather_rs"
    click a2a_transpose_gemm "https://github.com/bytedance/flux/tree/main/src/a2a_transpose_gemm"
    click coll "https://github.com/bytedance/flux/tree/main/src/coll"
    click comm_none "https://github.com/bytedance/flux/tree/main/src/comm_none"
    click inplace_cast "https://github.com/bytedance/flux/tree/main/src/inplace_cast"
    click quantization "https://github.com/bytedance/flux/tree/main/src/quantization"
    click Generator "https://github.com/bytedance/flux/tree/main/src/generator"
    click op_registry "https://github.com/bytedance/flux/blob/main/include/flux/op_registry.h"
    click op_registry_proto_utils "https://github.com/bytedance/flux/blob/main/include/flux/op_registry_proto_utils.h"
    click runtime_config "https://github.com/bytedance/flux/blob/main/include/flux/runtime_config.h"
    click dispatch_policy_ext "https://github.com/bytedance/flux/blob/main/include/flux/cuda/dispatch_policy_ext.hpp"
    click ths_op "https://github.com/bytedance/flux/blob/main/include/flux/ths_op/ths_op.h"
    click utils "https://github.com/bytedance/flux/blob/main/include/flux/utils.h"
    click flux_h "https://github.com/bytedance/flux/blob/main/include/flux/flux.h"
    click pybind "https://github.com/bytedance/flux/tree/main/src/pybind"
    click python_flux "https://github.com/bytedance/flux/tree/main/python/flux"
    click python_triton "https://github.com/bytedance/flux/tree/main/python/flux_triton"
    click CMakeLists "https://github.com/bytedance/flux/blob/main/CMakeLists.txt"
    click cmake "https://github.com/bytedance/flux/tree/main/cmake/"
    click build_sh "https://github.com/bytedance/flux/blob/main/build.sh"
    click install_deps "https://github.com/bytedance/flux/blob/main/install_deps.sh"
    click Dockerfile "https://github.com/bytedance/flux/tree/main/Dockerfile"
    click setup_py "https://github.com/bytedance/flux/blob/main/setup.py"
    click pyproject "https://github.com/bytedance/flux/blob/main/pyproject.toml"
    click setup_cfg "https://github.com/bytedance/flux/blob/main/setup.cfg"
    click manifest "https://github.com/bytedance/flux/blob/main/MANIFEST.in"
    click examples "https://github.com/bytedance/flux/tree/main/examples/"
    click test_py "https://github.com/bytedance/flux/tree/main/test/python/"
    click test_unit "https://github.com/bytedance/flux/tree/main/test/unit/"
    click test_tools "https://github.com/bytedance/flux/tree/main/test/tools/"
    click docs "https://github.com/bytedance/flux/tree/main/docs/"
    click readme "https://github.com/bytedance/flux/blob/main/README.md"

    %% Styles
    classDef hardware fill:#C0C0C0,stroke:#333,shape:parallelogram
    classDef thirdparty fill:#E0E0E0,stroke:#333,shape:cylinder
    classDef core fill:#ADD8E6,stroke:#333
    classDef build fill:#FFA500,stroke:#333
    classDef binding fill:#FFD700,stroke:#333
    classDef python fill:#90EE90,stroke:#333
```
