# huggingface/optimum-benchmark architecture

> Architecture diagram of the GitHub repository huggingface/optimum-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/huggingface/optimum-benchmark
- Repository: https://github.com/huggingface/optimum-benchmark

## Overview

This project is a benchmarking utility aimed at testing and measuring model performance (latency, throughput, memory, energy, etc.) for various Hugging Face libraries and frameworks (Transformers, Diffusers, PEFT, TIMM, and Optimum) across multiple backends (e.g. PyTorch, OnnxRuntime, OpenVINO, vLLM, TensorRT-LLM, IPEX, etc.) and devices (CPU, CUDA, ROCm, etc.). The system is designed to allow users to run benchmarks either through a Python API or using Hydra-driven CLI configurations, and it supports both isolated and distributed execution (via multiple launchers).

##### Main Components
- Backends: Each backend (pytorch, onnxruntime, openvino, etc.) is implemented under the “optimum_benchmark/backends” directory. They include specific configuration and utility files to perform benchmarking on the targeted framework/device.
- Scenarios: The project differentiates between scenarios such as “inference” and “training”. These are found under “optimum_benchmark/scenarios” (and also there is a special “energy_star” scenario set in its dedicated directory).
- Launchers: Different execution modes exist (process, torchrun, inline); these are defined under “optimum_benchmark/launchers” with further subdivision for process isolation or distributed setups.
- Benchmark Reporting & Tracking: Modules for tracking runtime metrics (latency, memory, energy) are part of the core design, along with tools for generating and exporting reports (JSON, CSV, markdown).

##### Supporting Components
- Configuration Modules – multiple YAML and Python configuration examples are provided in the “examples” and “tests/configs” directories. These define how benchmarks are set up (e.g., backend, scenario, launcher configurations using Hydra).
- CLI and Python API Interface – there’s an easy-to-use CLI as well as a Python-based API that allows users to programmatically run benchmarks. This is critical for integration into different workflows.
- CI/CD & Docker – the repository contains a series of GitHub workflow files (under “.github/workflows”) for automated testing, quality checks, and security. Also, there are Docker files in the “docker” directory that encapsulate the environment required for different hardware and software setups.
- External Services & Hub Integration – functionality is provided to push benchmark results to the Hugging Face hub, integrating with external services.

##### Relationships and Data Flow
- Start at the interface layer: User Initiates a Benchmark via the CLI or Python API.
- The configuration (YAML examples or Python configuration objects) is read to instantiate a BenchmarkConfig.
- This object is then passed to the Benchmark Engine where it orchestrates:
- Selection and instantiation of the appropriate launcher (process, torchrun, etc.).
- Setting up the chosen scenario (inference or training) along with its input specifications.
- Invoking the suitable backend, which is responsible for model loading (or initialization without weights) and performing the actual computation.
- During execution, the benchmark’s trackers (for latency, memory, energy, etc.) continuously capture metrics.
- Post benchmark execution, a BenchmarkReport is generated, exported (as JSON/CSV), logged, and optionally pushed to the Hugging Face hub.

##### Architectural Patterns and Design Principles
- Modular Design: Clear separation between backends, scenarios, and launchers to allow easy extension (adding new backends or scenarios is straightforward).
- Configuration as Code: Heavy reliance on structured configuration files (YAML) powered by Hydra, keeping the system flexible and easily customizable.
- Scalability & Extensibility: The design supports both isolated and distributed benchmarking, showing consideration for scaling across multiple devices or nodes.
- CI/CD Integration: Automated pipelines ensure that every change is validated across multiple configurations and hardware targets.
- Containerization: Docker files indicate that containerized benchmarking is supported, centralizing environment management.

##### Instructing the Principal Software Engineer
- Adding annotations that indicate important features such as “Device selection (cpu, cuda, rocm)”, “Hydra-driven configuration”, or “Docker containerization for reproducibility”.
- To annotate the modularity by indicating that each “backend” has its own submodules and configuration file, which may be extended independently.

Happy diagramming!

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User Interface Layer"
        CLI["CLI & Python API"]:::ui
    end

    subgraph "Configuration Layer"
        Config1["Examples Config"]:::config
        Config2["Test Configs"]:::config
    end

    subgraph "Core Benchmark Engine"
        Engine["Benchmark Engine"]:::core
        Launcher["Launchers"]:::core
        Scenario["Scenarios (Inference/Training)"]:::core
        Energy["Energy Star Scenario"]:::core
        Backends["Backends"]:::core
        Report["Benchmark Report (JSON/CSV)"]:::core
        Trackers["Metric Trackers"]:::core
    end

    subgraph "External Integration Layer"
        CI["CI/CD Workflows"]:::external
        Docker["Docker Environment"]:::external
        Hub["HuggingFace Hub Integration"]:::external
    end

    %% Flow Connections
    CLI -->|"initiates"| Config1
    CLI -->|"initiates"| Config2
    Config1 -->|"parses config"| Engine
    Config2 -->|"parses config"| Engine
    Engine -->|"selects"| Launcher
    Engine -->|"sets up"| Scenario
    Engine -->|"sets up"| Energy
    Engine -->|"invokes"| Backends
    Engine -->|"captures metrics"| Trackers
    Engine -->|"generates"| Report
    Report -->|"exports to"| Hub
    Trackers -->|"feeds data to"| Hub

    %% Click Events
    click CLI "https://github.com/huggingface/optimum-benchmark/blob/main/optimum_benchmark/cli.py"
    click Config1 "https://github.com/huggingface/optimum-benchmark/tree/main/examples"
    click Config2 "https://github.com/huggingface/optimum-benchmark/tree/main/tests/configs"
    click Engine "https://github.com/huggingface/optimum-benchmark/tree/main/optimum_benchmark/benchmark"
    click Launcher "https://github.com/huggingface/optimum-benchmark/tree/main/optimum_benchmark/launchers"
    click Scenario "https://github.com/huggingface/optimum-benchmark/tree/main/optimum_benchmark/scenarios"
    click Energy "https://github.com/huggingface/optimum-benchmark/tree/main/energy_star"
    click Backends "https://github.com/huggingface/optimum-benchmark/tree/main/optimum_benchmark/backends"
    click Report "https://github.com/huggingface/optimum-benchmark/blob/main/optimum_benchmark/benchmark/report.py"
    click Trackers "https://github.com/huggingface/optimum-benchmark/tree/main/optimum_benchmark/trackers"
    click CI "https://github.com/huggingface/optimum-benchmark/tree/main/.github/workflows"
    click Docker "https://github.com/huggingface/optimum-benchmark/tree/main/docker"
    click Hub "https://github.com/huggingface/optimum-benchmark/blob/main/optimum_benchmark/hub_utils.py"

    %% Styling Classes
    classDef ui fill:#a2d2ff,stroke:#000,stroke-width:2px;
    classDef config fill:#caffbf,stroke:#000,stroke-width:2px;
    classDef core fill:#ffadad,stroke:#000,stroke-width:2px;
    classDef external fill:#ffd6a5,stroke:#000,stroke-width:2px;
```
