How rocm/aiter works
Project type and purpose
• “aiter” is an open-source, high-performance AI operator library.
• It exposes both Python-level and C++-level APIs and bundles multiple GPU kernel back-ends (Triton, CK, ASM, HSA).
• It also provides training, inference, GEMM and distributed-communication primitives, tuners, and configuration files.
Top-level components (from file tree and README)
– aiter/ops, aiter/bert_padding, aiter/paged_attn, etc.: Python operator definitions and dispatch logic
– aiter/jit: runtime compilation pipeline (core.py, hipify, C++ extension versioning)
– aiter/dist: distributed-communication primitives (all-reduce, broadcast, parallel state)
– aiter/configs: CSV files driving kernel selection and auto-tuning
– csrc/: C++/CUDA/HIP sources, kernels, headers, pybind wrappers, and interface generators
– hsa/: pre-compiled HSA (AMD GPU) code objects for low-level kernels
– 3rdparty/composable_kernel: external kernel library submodule
– gradlib/: standalone gradient tuner library
– op_benchmarks/ and op_tests/: benchmarking and test suites driving coverage
– setup.py & requirements.txt: build/install scripts
Key architectural layers and relationships
a. Python API layer
• User code imports aiter.<op> modules (e.g. aiter.ops.attention).
b. JIT & build pipeline
• aiter/jit/core.py orchestrates on-the-fly compilation, uses hipify to translate CUDA→HIP, and invokes cpp_extension utilities.
• Results in building C++ extension modules (via setup.py develop).
c. C++ extension / PyBind layer
• csrc/pybind/*.cu define PYBIND11_MODULE entry points.
• On import, these modules register C++ operator entry points into the Python runtime.
d. Kernel library
• csrc/kernels/*.cu – hand-written CUDA/HIP kernel implementations
• 3rdparty/composable_kernel – templated CK kernels
• csrc/ck_*/ & hsa/ – tuned code objects for specific data formats and hardware
e. Distributed communication
• aiter/dist implements custom all-reduce, shared-memory broadcast, etc., on top of HSA/hip.
f. Configuration & tuning
• CSV files in aiter/configs and JSON in jit/optCompilerConfig drive which kernel variant to use.
• gradlib provides autotuning for GEMM parameters.