# rocm/aiter architecture

> Architecture diagram of the GitHub repository rocm/aiter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rocm/aiter
- Repository: https://github.com/rocm/aiter

## Overview

##### Project type and purpose
- “aiter” is an open-source, high-performance AI operator library.
- It exposes both Python-level and C++-level APIs and bundles multiple GPU kernel back-ends (Triton, CK, ASM, HSA).
- It also provides training, inference, GEMM and distributed-communication primitives, tuners, and configuration files.

##### Top-level components (from file tree and README)
– aiter/ops, aiter/bert_padding, aiter/paged_attn, etc.: Python operator definitions and dispatch logic
– aiter/jit: runtime compilation pipeline (core.py, hipify, C++ extension versioning)
– aiter/dist: distributed-communication primitives (all-reduce, broadcast, parallel state)
– aiter/configs: CSV files driving kernel selection and auto-tuning
– csrc/: C++/CUDA/HIP sources, kernels, headers, pybind wrappers, and interface generators
– hsa/: pre-compiled HSA (AMD GPU) code objects for low-level kernels
– 3rdparty/composable_kernel: external kernel library submodule
– gradlib/: standalone gradient tuner library
– op_benchmarks/ and op_tests/: benchmarking and test suites driving coverage
– setup.py & requirements.txt: build/install scripts

##### Key architectural layers and relationships
a. Python API layer
- User code imports aiter.<op> modules (e.g. aiter.ops.attention).
b. JIT & build pipeline
- aiter/jit/core.py orchestrates on-the-fly compilation, uses hipify to translate CUDA→HIP, and invokes cpp_extension utilities.
- Results in building C++ extension modules (via setup.py develop).
c. C++ extension / PyBind layer
- csrc/pybind/*.cu define `PYBIND11_MODULE` entry points.
- On import, these modules register C++ operator entry points into the Python runtime.
d. Kernel library
- csrc/kernels/*.cu – hand-written CUDA/HIP kernel implementations
- 3rdparty/composable_kernel – templated CK kernels
- csrc/ck_*/ & hsa/ – tuned code objects for specific data formats and hardware
e. Distributed communication
- aiter/dist implements custom all-reduce, shared-memory broadcast, etc., on top of HSA/hip.
f. Configuration & tuning
- CSV files in aiter/configs and JSON in jit/optCompilerConfig drive which kernel variant to use.
- gradlib provides autotuning for GEMM parameters.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Python Frontend Layer
    subgraph "Python Frontend" 
        direction TB
        PF1["Operator Definitions & Dispatch"]:::python
        PF2["bert_padding.py"]:::python
        PF3["paged_attn.py"]:::python
        PF4["rotary_embedding.py"]:::python
        PF5["tuned_gemm.py"]:::python
        PF6["fused_moe.py"]:::python
        PF7["fused_moe_bf16_asm.py"]:::python
    end

    %% Configuration & Tuning Layer
    subgraph "Configuration & Tuning" 
        direction TB
        CFG["CSV Configs"]:::config
    end

    %% Autotuning Library
    subgraph "Autotuning Library" 
        direction TB
        GT1["GemmTuner.py"]:::config
        GT2["grad_funcs.cu"]:::config
        GT3["setup.py"]:::config
    end

    %% JIT & Build Pipeline
    subgraph "JIT & Build Pipeline"
        direction TB
        JIT1["core.py"]:::jit
        JIT2["hipify_python.py"]:::jit
        JIT3["_cpp_extension_versioner.py"]:::jit
        JIT4["optCompilerConfig.json"]:::config
    end

    %% C++ Extension Layer
    subgraph "C++ Extension Layer"
        direction TB
        CE["pybind Modules"]:::cpp
    end

    %% Kernel Library
    subgraph "Kernel Library"
        direction TB
        KL1["Hand-Written Kernels"]:::kernel
        KL2["Composable Kernel Submodule"]:::kernel
        KL3["CK Backends & Tuners"]:::kernel
        KL4["Pre-Compiled HSA Objects"]:::kernel
    end

    %% Distributed Communication
    subgraph "Distributed Communication"
        direction TB
        DC1["communication_op.py"]:::dist
        DC2["custom_all_reduce.py"]:::dist
        DC3["shm_broadcast.py"]:::dist
        DC4["parallel_state.py"]:::dist
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        ED1["ROCm/HIP Runtime"]:::external
        ED2["Triton"]:::external
        ED3["PyTorch"]:::external
    end

    %% Testing & Benchmarking
    subgraph "Testing & Benchmarking"
        direction TB
        BM["op_benchmarks"]:::testing
        TS1["op_tests"]:::testing
        TS2["op_tests/triton_tests"]:::testing
    end

    %% Build & Install Scripts
    subgraph "Build & Install Scripts"
        direction TB
        BS1["setup.py"]:::build
        BS2["requirements.txt"]:::build
    end

    %% Connections
    PF1 -->|"selects kernel"| CFG
    PF1 -->|"uses autotuning"| GT1
    PF1 -->|"dispatches"| CE
    CFG -.->|"drives variant"| KL1
    CFG -.->|"drives variant"| KL2
    GT1 -->|"tunes parameters"| KL3
    PF1 -->|"if not compiled: compile"| JIT1
    JIT1 -->|"uses hipify"| JIT2
    JIT1 -->|"version check"| JIT3
    JIT1 -->|"reads config"| JIT4
    JIT1 -->|"builds extension"| CE
    CE -->|"launch kernel"| KL1
    CE -->|"launch CK kernels"| KL2
    CE -->|"launch HSA kernels"| KL4
    CE -->|"calls all-reduce"| DC1
    DC1 -->|"communicates via HIP"| ED1
    KL1 -->|"executes on GPU"| ED1
    KL2 -->|"executes on GPU"| ED1
    KL4 -->|"executes on GPU"| ED1
    BM -->|"benchmarks ops"| PF1
    TS1 -->|"tests Python"| PF1
    TS2 -->|"tests Triton backends"| ED2
    BS1 --> JIT1
    BS2 --> PF1

    %% Click Events
    click PF1 "https://github.com/rocm/aiter/tree/main/aiter/ops/"
    click PF2 "https://github.com/rocm/aiter/blob/main/aiter/bert_padding.py"
    click PF3 "https://github.com/rocm/aiter/blob/main/aiter/paged_attn.py"
    click PF4 "https://github.com/rocm/aiter/blob/main/aiter/rotary_embedding.py"
    click PF5 "https://github.com/rocm/aiter/blob/main/aiter/tuned_gemm.py"
    click PF6 "https://github.com/rocm/aiter/blob/main/aiter/fused_moe.py"
    click PF7 "https://github.com/rocm/aiter/blob/main/aiter/fused_moe_bf16_asm.py"
    click JIT1 "https://github.com/rocm/aiter/blob/main/aiter/jit/core.py"
    click JIT2 "https://github.com/rocm/aiter/blob/main/aiter/jit/utils/hipify/hipify_python.py"
    click JIT3 "https://github.com/rocm/aiter/blob/main/aiter/jit/utils/_cpp_extension_versioner.py"
    click JIT4 "https://github.com/rocm/aiter/blob/main/aiter/jit/optCompilerConfig.json"
    click DC1 "https://github.com/rocm/aiter/blob/main/aiter/dist/communication_op.py"
    click DC2 "https://github.com/rocm/aiter/blob/main/aiter/dist/custom_all_reduce.py"
    click DC3 "https://github.com/rocm/aiter/blob/main/aiter/dist/shm_broadcast.py"
    click DC4 "https://github.com/rocm/aiter/blob/main/aiter/dist/parallel_state.py"
    click CFG "https://github.com/rocm/aiter/tree/main/aiter/configs/"
    click CE "https://github.com/rocm/aiter/tree/main/csrc/pybind/"
    click KL1 "https://github.com/rocm/aiter/tree/main/csrc/kernels/"
    click KL2 "https://github.com/rocm/aiter/tree/main/3rdparty/composable_kernel/"
    click KL3 "https://github.com/rocm/aiter/tree/main/csrc/ck_batched_gemm_a8w8/"
    click KL4 "https://github.com/rocm/aiter/tree/main/hsa/"
    click GT1 "https://github.com/rocm/aiter/blob/main/gradlib/gradlib/GemmTuner.py"
    click GT2 "https://github.com/rocm/aiter/blob/main/gradlib/csrc/grad_funcs.cu"
    click GT3 "https://github.com/rocm/aiter/blob/main/gradlib/setup.py"
    click BM "https://github.com/rocm/aiter/tree/main/op_benchmarks/"
    click TS1 "https://github.com/rocm/aiter/tree/main/op_tests/"
    click TS2 "https://github.com/rocm/aiter/tree/main/op_tests/triton_tests/"
    click BS1 "https://github.com/rocm/aiter/blob/main/setup.py"
    click BS2 "https://github.com/rocm/aiter/blob/main/requirements.txt"

    %% Styles
    classDef python fill:#ADD8E6,stroke:#000
    classDef jit fill:#90EE90,stroke:#000
    classDef cpp fill:#FFA500,stroke:#000
    classDef kernel fill:#FF6347,stroke:#000
    classDef dist fill:#DDA0DD,stroke:#000
    classDef config fill:#FFFF99,stroke:#000
    classDef external fill:#D3D3D3,stroke:#000
    classDef testing fill:#E6E6FA,stroke:#000
    classDef build fill:#FFE4B5,stroke:#000
```
