How rocm/rccl-tests works
Main Components
• Build & Configuration System:
• Build scripts (Makefile at root, CMakeLists.txt in root and under “src”) and installation scripts (install.sh).
• Configuration for different environments: HIP, MPI, RCCL is configurable via environment variables (e.g., HIP_HOME, NCCL_HOME, MPI_HOME).
• Core Test Implementations:
• The “src” directory contains the core C/CUDA source files (e.g., all_reduce.cu, broadcast.cu, alltoall.cu, etc.) which implement different collective operations.
• “verifiable” directory holds additional tests or verification components (e.g., verifiable.cu, verifiable.h, Makefile) used to verify correctness.
• Automated Unit & Integration Testing:
• The “test” directory houses Python unit tests (using pytest) that likely run higher-level correctness checks.
• CI/CD & Scripting:
• Directories like “.github” and “.jenkins” contain CI configuration files (GitHub workflows, Jenkins groovy scripts) that integrate continuous testing and static analysis into the development process.
• Docker Environment:
• Dockerfiles in the “docker” directory show that containerized builds/installations are supported, targeting different platforms (Centos, Ubuntu).
Relationships and Interactions
• The build system (Make/CMake) compiles the CUDA-based tests from the “src” folder and also may handle the “verifiable” components.
• The test runners (both binary executables built from “src” and the Python tests in “test”) are executed either locally or via MPI in multi-GPU environments.
• CI/CD tools (Jenkins, GitHub Actions) trigger builds and tests automatically upon code changes.
• Docker environments provide isolated and reproducible build and run contexts to ensure consistency across different platforms.
• External dependencies like HIP, MPI, and RCCL are injected at build time via environmental variables and compiler configurations.
Important Architectural Patterns and Design Principles
• Separation of Concerns: Differentiation between core operations (in “src”), verification (“verifiable”), unit tests (“test”), and deployment/integration (CI/CD scripts and Dockerfiles).
• Configurability: The build scripts and README emphasize tailoring builds via specific environment variables and flags (e.g., MPI support, HIP paths).
• Modular Testing: The design supports both performance and functional correctness testing—each collective operation is individually implemented and then tested.
• Containerization and Continuous Integration: The existence of Docker and CI configuration promotes reproducibility and automated testing.
Relevant Technologies, Frameworks, or Libraries
• Programming Languages: C/CUDA (in .cu and .cc files), Python for testing.
• Build Tools: Make, CMake.
• CI/CD: Jenkins and GitHub Actions.
• Containerization: Docker.
• Dependencies: HIP, MPI, RCCL. Additionally, frameworks like pytest for unit testing.
• Performance Benchmarking: Custom command-line argument parsing to set testing parameters (number of GPUs, threads, message sizes, etc.).