How pku-dair/hetu works
Main Components to Identify
• Frontend & High-Level API Layer: This is mainly the Python interface (located under directories like “python/hetu/…”) that exposes user-facing APIs and constructs the computational or operator graph. It also contains high-level distributed strategies and model definitions.
• Graph & Execution Engine: The core C++ components (found in “hetu/graph”, “hetu/core”, “hetu/impl”) maintain the building and execution of the computational graph. This is where operators are defined, scheduled, and lowered to kernels.
• Kernel & Operator Library: There are many operator implementations in C++ and CUDA (in “hetu/impl/kernel”, “hetu/ops”, “hetu/v1/src/ops”), which provide the actual mathematical operations on CPU/GPU.
• Memory Management: Modules such as “hetu/impl/memory” implement low-level memory pools and allocation schemes required for efficient training on GPUs (with specialized memory management for mixed precision, tensor splitting, etc.).
• Communication & Distributed Runtime: The distributed components are responsible for communication (using MPI, NCCL, gRPC) as seen in “hetu/impl/communication” and “python/hetu/_binding/distributed”. These enable multi-node and multi-GPU coordination.
• Scheduling & Parallel Strategies: The engine also includes scheduling strategies (for pipeline, data parallelism, model parallelism, and hybrid approaches) evident through components in the Python distributed strategies and configuration files in multiple “ds_parallel_config” directories as well as graph scheduling in the C++ code base.
• Build & Configuration Infrastructure: The CMake modules (in “cmake/…”) form the build system that configures, enables/disables modules, and links external dependencies (like CUDA, CuDNN, NCCL, PyBind11, gRPC, etc.).
• Tools, Tests & Profilers: Additional directories (e.g., “tests”, “tools”, “examples”) contain scripts for benchmarking, debugging, profiling (e.g., the profiling of operator kernels and communication costs) and provide examples to validate the design and performance.
Relationships and Interactions
• The Python frontend acts as the entry point: user code (model definitions, training scripts) calls into the Python API, which then builds a computational graph.
• The graph execution engine (in C++) receives this graph and schedules the computation over available devices. It maps high-level tensor operations to optimized kernels.
• Kernel implementations (C++/CUDA) perform the low-level computations. They call upon memory management routines to allocate buffers and then they use the communication layer to synchronize data between different nodes.
• The communication layer is built on top of external libraries (MPI, NCCL) and is responsible for operations like parameter synchronization, all-reduce operations, and distributed gradient exchange.
• The scheduling modules decide on the parallel strategy, whether it’s data, model, pipeline, or hybrid parallelism, and orchestrate the overall execution workflow across devices.
• The build system and configuration files allow tailoring the system: users enable or disable certain modules (like specific operator kernels or communication backends) and tune system parameters.
• Tools and tests interface with all modules to benchmark performance and verify correctness.
Implementation Steps
b. Subdivide it horizontally or vertically into layers:
• Top Layer: Python Frontend (API, distributed strategies, model definitions)
• Middle Layer: Graph Engine & Scheduler (graph compilation, operator scheduling)
• Lower Layer: Kernel Library & Memory Management (optimized kernels, allocation and memory pooling)
• Side/Adjacent Components: Distributed Communication Modules (MPI, NCCL, gRPC) and Build/Configuration Infrastructure.
• Auxiliary: Tools, Tests, and Profilers.