# pku-dair/hetu architecture

> Architecture diagram of the GitHub repository pku-dair/hetu, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pku-dair/hetu
- Repository: https://github.com/pku-dair/hetu

## Overview

##### Main Components to Identify
- Frontend & High-Level API Layer: This is mainly the Python interface (located under directories like “python/hetu/…”) that exposes user-facing APIs and constructs the computational or operator graph. It also contains high-level distributed strategies and model definitions.
- Graph & Execution Engine: The core C++ components (found in “hetu/graph”, “hetu/core”, “hetu/impl”) maintain the building and execution of the computational graph. This is where operators are defined, scheduled, and lowered to kernels.
- Kernel & Operator Library: There are many operator implementations in C++ and CUDA (in “hetu/impl/kernel”, “hetu/ops”, “hetu/v1/src/ops”), which provide the actual mathematical operations on CPU/GPU.
- Memory Management: Modules such as “hetu/impl/memory” implement low-level memory pools and allocation schemes required for efficient training on GPUs (with specialized memory management for mixed precision, tensor splitting, etc.).
- Communication & Distributed Runtime: The distributed components are responsible for communication (using MPI, NCCL, gRPC) as seen in “hetu/impl/communication” and “python/hetu/_binding/distributed”. These enable multi-node and multi-GPU coordination.
- Scheduling & Parallel Strategies: The engine also includes scheduling strategies (for pipeline, data parallelism, model parallelism, and hybrid approaches) evident through components in the Python distributed strategies and configuration files in multiple “ds_parallel_config” directories as well as graph scheduling in the C++ code base.
- Build & Configuration Infrastructure: The CMake modules (in “cmake/…”) form the build system that configures, enables/disables modules, and links external dependencies (like CUDA, CuDNN, NCCL, PyBind11, gRPC, etc.).
- Tools, Tests & Profilers: Additional directories (e.g., “tests”, “tools”, “examples”) contain scripts for benchmarking, debugging, profiling (e.g., the profiling of operator kernels and communication costs) and provide examples to validate the design and performance.

##### Relationships and Interactions
- The Python frontend acts as the entry point: user code (model definitions, training scripts) calls into the Python API, which then builds a computational graph.
- The graph execution engine (in C++) receives this graph and schedules the computation over available devices. It maps high-level tensor operations to optimized kernels.
- Kernel implementations (C++/CUDA) perform the low-level computations. They call upon memory management routines to allocate buffers and then they use the communication layer to synchronize data between different nodes.
- The communication layer is built on top of external libraries (MPI, NCCL) and is responsible for operations like parameter synchronization, all-reduce operations, and distributed gradient exchange.
- The scheduling modules decide on the parallel strategy, whether it’s data, model, pipeline, or hybrid parallelism, and orchestrate the overall execution workflow across devices.
- The build system and configuration files allow tailoring the system: users enable or disable certain modules (like specific operator kernels or communication backends) and tune system parameters.
- Tools and tests interface with all modules to benchmark performance and verify correctness.

##### Implementation Steps
b. Subdivide it horizontally or vertically into layers:
- Top Layer: Python Frontend (API, distributed strategies, model definitions)
- Middle Layer: Graph Engine & Scheduler (graph compilation, operator scheduling)
- Lower Layer: Kernel Library & Memory Management (optimized kernels, allocation and memory pooling)
- Side/Adjacent Components: Distributed Communication Modules (MPI, NCCL, gRPC) and Build/Configuration Infrastructure.
- Auxiliary: Tools, Tests, and Profilers.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Hetu Distributed Deep Learning System"]:::system

    subgraph "Frontend & High-Level API Layer"
        FE1["Python Frontend API"]:::python
        FE2["Example Training Scripts"]:::python
    end

    subgraph "Graph & Execution Engine"
        GE1["Graph Construction & Execution Engine"]:::cpp
        GE2["C++ Graph Bindings"]:::cpp
        S1["Distributed Strategies & Trainer"]:::scheduler
        S2["Parallel Configurations"]:::scheduler
    end

    subgraph "Kernel & Operator Library and Memory Management"
        K1["Kernel & Operator Library"]:::cpp
        O1["Operator Definitions"]:::cpp
        MEM["Memory Pool Manager"]:::memory
    end

    subgraph "Distributed Communication Modules"
        COM1["Communication Module"]:::comm
        COM2["Distributed Runtime Bindings"]:::comm
    end

    subgraph "Build & Configuration Infrastructure"
        BC1["Build Config (CMake)"]:::build
        BC2["CMake Modules"]:::build
    end

    subgraph "Tools, Tests & Profilers"
        TT1["Benchmark & Profiler Tools"]:::tools
        TT2["Unit/Integration Tests"]:::tools
        TT3["Example Workflows"]:::tools
    end

    A --> FE1
    FE1 -->|"builds_graph"| GE1
    FE2 -->|"supplies_examples"| FE1

    GE1 -->|"binds_graph"| GE2
    GE1 -->|"triggers_schedule"| S1
    S2 -->|"configures_parallel"| S1

    GE1 -->|"dispatches_kernels"| K1
    GE2 --> K1

    K1 -->|"calls_operator"| O1
    K1 -->|"allocates_memory"| MEM

    K1 -->|"synchronizes"| COM1
    COM1 -->|"binds_comm"| COM2

    BC1 ---|"configures"| FE1
    BC1 ---|"configures"| GE1
    BC1 ---|"configures"| K1
    BC2 ---|"links"| COM1

    TT1 ---|"benchmarks"| FE1
    TT1 ---|"benchmarks"| GE1
    TT2 ---|"tests"| K1
    TT3 ---|"demonstrates"| FE1

    FE1 --> GE1
    GE1 --> K1
    K1 --> COM1
    COM1 --> BC1
    BC1 --> TT1

    classDef system fill:#f9f,stroke:#333,stroke-width:2px;
    classDef python fill:#bbf,stroke:#333,stroke-width:2px;
    classDef cpp fill:#bfb,stroke:#333,stroke-width:2px;
    classDef scheduler fill:#ff9,stroke:#333,stroke-width:2px;
    classDef memory fill:#f99,stroke:#333,stroke-width:2px;
    classDef comm fill:#9cf,stroke:#333,stroke-width:2px;
    classDef build fill:#fc9,stroke:#333,stroke-width:2px;
    classDef tools fill:#cff,stroke:#333,stroke-width:2px;

    click FE1 "https://github.com/pku-dair/hetu/tree/main/python/hetu"
    click FE2 "https://github.com/pku-dair/hetu/tree/main/examples/hetu"
    click GE1 "https://github.com/pku-dair/hetu/tree/main/hetu/graph"
    click GE2 "https://github.com/pku-dair/hetu/tree/main/python/hetu/_binding/graph"
    click K1 "https://github.com/pku-dair/hetu/tree/main/hetu/impl/kernel"
    click O1 "https://github.com/pku-dair/hetu/tree/main/hetu/graph/ops"
    click MEM "https://github.com/pku-dair/hetu/tree/main/hetu/impl/memory"
    click COM1 "https://github.com/pku-dair/hetu/tree/main/hetu/impl/communication"
    click COM2 "https://github.com/pku-dair/hetu/tree/main/python/hetu/_binding/distributed"
    click S1 "https://github.com/pku-dair/hetu/tree/main/python/hetu/engine"
    click S2 "https://github.com/pku-dair/hetu/tree/main/tests/ci_test/ds_parallel_config"
    click BC1 "https://github.com/pku-dair/hetu/blob/main/CMakeLists.txt"
    click BC2 "https://github.com/pku-dair/hetu/tree/main/cmake/"
    click TT1 "https://github.com/pku-dair/hetu/tree/main/tools"
    click TT2 "https://github.com/pku-dair/hetu/tree/main/tests"
    click TT3 "https://github.com/pku-dair/hetu/tree/main/examples"
```
