How bytedance/flux works
Project Type and Purpose
• Flux is a high-performance, fine-grained GPU kernel library with Python bindings for PyTorch.
• It fuses computation and communication (dense GEMM, MoE kernels, all-gather, reduce-scatter, etc.) to maximize GPU utilization and overlap.
Main Components
a. Third-Party Dependencies
• 3rdparty/CUTLASS (tensor-core GEMM templates)
• 3rdparty/NCCL (multi-GPU collectives)
• NVSHMEM (optional, for MoE kernels)
b. Core C++/CUDA Kernel Modules (src/…)
• gemm_only, ag_gemm, gemm_rs, moe_ag_scatter, moe_gather_rs, a2a_transpose_gemm, coll, comm_none, inplace_cast, quantization
• Each sub-folder contains:
– CUDA kernel implementations (.cu/.cuh)
– Host-side glue (ths_op)
– Tuning configurations for different GPU architectures
c. Code Generator (src/generator/)
• Reads high-level descriptions and emits specialized C++ kernels for each submodule
d. Public API Headers (include/flux/)
• op_registry, runtime_config, utils, dispatch policies
• These define the compile-time and run-time interfaces
e. Python Bindings
• C++ extension (src/pybind/ using pybind11)
• Python packages: python/flux (user API), python/flux_triton (Triton backend)
f. Build & Packaging
• CMake (CMakeLists.txt, cmake/)
• build.sh, Dockerfile, install_deps.sh, setup.py, pyproject.toml
• Produces: shared libraries, Python wheel
g. User Layer
• Examples (examples/)
• Tests (test/python/, test/unit/, test/tools/)
• Documentation (docs/, README.md)
Component Relationships & Data Flow
• Code Generator → emits C++ kernel source → compiled by Build System
• Build System (CMake/build.sh) → compiles 3rdparty + kernel modules → links into shared libs + Python extension
• Core C++ modules register kernels via op_registry
• PyBind extension loads C++ modules → exposes Flux operations to Python
• Python flux API calls into extension → launches fused CUDA kernels
• Kernels coordinate via NCCL/NVSHMEM for inter-GPU communication and CUTLASS for compute
• User scripts/tests/examples invoke Python API → drive GPU workloads
Architectural Patterns & Design Principles
• Modular submodule structure: one folder per kernel family, consistent layout
• Plugin-style op registry for decoupled implementation and dispatch
• Build-time code generation for specialized kernels
• Multi-language layering: C++/CUDA core + Python wrapper
• Separation of concerns: compute, communication, configuration, binding