How huggingface/optimum-benchmark works
This project is a benchmarking utility aimed at testing and measuring model performance (latency, throughput, memory, energy, etc.) for various Hugging Face libraries and frameworks (Transformers, Diffusers, PEFT, TIMM, and Optimum) across multiple backends (e.g. PyTorch, OnnxRuntime, OpenVINO, vLLM, TensorRT-LLM, IPEX, etc.) and devices (CPU, CUDA, ROCm, etc.). The system is designed to allow users to run benchmarks either through a Python API or using Hydra-driven CLI configurations, and it supports both isolated and distributed execution (via multiple launchers).
Main Components
• Backends: Each backend (pytorch, onnxruntime, openvino, etc.) is implemented under the “optimum_benchmark/backends” directory. They include specific configuration and utility files to perform benchmarking on the targeted framework/device.
• Scenarios: The project differentiates between scenarios such as “inference” and “training”. These are found under “optimum_benchmark/scenarios” (and also there is a special “energy_star” scenario set in its dedicated directory).
• Launchers: Different execution modes exist (process, torchrun, inline); these are defined under “optimum_benchmark/launchers” with further subdivision for process isolation or distributed setups.
• Benchmark Reporting & Tracking: Modules for tracking runtime metrics (latency, memory, energy) are part of the core design, along with tools for generating and exporting reports (JSON, CSV, markdown).
Supporting Components
• Configuration Modules – multiple YAML and Python configuration examples are provided in the “examples” and “tests/configs” directories. These define how benchmarks are set up (e.g., backend, scenario, launcher configurations using Hydra).
• CLI and Python API Interface – there’s an easy-to-use CLI as well as a Python-based API that allows users to programmatically run benchmarks. This is critical for integration into different workflows.
• CI/CD & Docker – the repository contains a series of GitHub workflow files (under “.github/workflows”) for automated testing, quality checks, and security. Also, there are Docker files in the “docker” directory that encapsulate the environment required for different hardware and software setups.
• External Services & Hub Integration – functionality is provided to push benchmark results to the Hugging Face hub, integrating with external services.
Relationships and Data Flow
• Start at the interface layer: User Initiates a Benchmark via the CLI or Python API.
• The configuration (YAML examples or Python configuration objects) is read to instantiate a BenchmarkConfig.
• This object is then passed to the Benchmark Engine where it orchestrates:
• Selection and instantiation of the appropriate launcher (process, torchrun, etc.).
• Setting up the chosen scenario (inference or training) along with its input specifications.
• Invoking the suitable backend, which is responsible for model loading (or initialization without weights) and performing the actual computation.
• During execution, the benchmark’s trackers (for latency, memory, energy, etc.) continuously capture metrics.
• Post benchmark execution, a BenchmarkReport is generated, exported (as JSON/CSV), logged, and optionally pushed to the Hugging Face hub.
Architectural Patterns and Design Principles
• Modular Design: Clear separation between backends, scenarios, and launchers to allow easy extension (adding new backends or scenarios is straightforward).
• Configuration as Code: Heavy reliance on structured configuration files (YAML) powered by Hydra, keeping the system flexible and easily customizable.
• Scalability & Extensibility: The design supports both isolated and distributed benchmarking, showing consideration for scaling across multiple devices or nodes.
• CI/CD Integration: Automated pipelines ensure that every change is validated across multiple configurations and hardware targets.
• Containerization: Docker files indicate that containerized benchmarking is supported, centralizing environment management.
Instructing the Principal Software Engineer
• Adding annotations that indicate important features such as “Device selection (cpu, cuda, rocm)”, “Hydra-driven configuration”, or “Docker containerization for reproducibility”.
• To annotate the modularity by indicating that each “backend” has its own submodules and configuration file, which may be extended independently.
Happy diagramming!