# cloudmercato/ai-benchmark architecture

> Architecture diagram of the GitHub repository cloudmercato/ai-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cloudmercato/ai-benchmark
- Repository: https://github.com/cloudmercato/ai-benchmark

## Overview

##### Project Type and Purpose
- This is an **open-source Python library + CLI tool** for measuring inference/training performance of deep-learning models on various hardware (CPUs, GPUs, TPUs).
- Key technologies: Python, TensorFlow 1.x/2.x, Docker, CUDA/cuDNN.

##### File Structure
Top-level components:
- bin/ai-benchmark – CLI entry-point script
- ai_benchmark/ – core Python package
– config.py – definitions of tests, sections, default parameters
– console.py – command-line interface wrapper around the core
– core.py – AIBenchmark class orchestrating test runs
– data/ – bundled small sample datasets for classification, segmentation, NLP, etc.
– model_utils.py – loading TensorFlow meta-graphs, building sessions
– models/ .meta files – pre-exported TensorFlow graph definitions
– update_utils.py – functionality to download or update model files
– utils.py – helper functions (logging, result formatting)
– tests/ – unit and functional tests for each module
- Dockerfile* – container definitions for CPU, GPU, PPC64LE builds
- .travis.yml, tox.ini – CI configurations
- setup.py, setup.cfg, MANIFEST.in – packaging & distribution

##### Readme Insights
- 42 individual tests across 19 sections
- Exposes high-level API (AIBenchmark.run(), .run_inference(), .run_training())
- CLI interface via `ai-benchmark` console command
- Supports advanced settings: CPU vs GPU, verbosity, precision

##### Main Components & Their Interactions
a. **CLI Layer** (bin/ai-benchmark + console.py)
- Parses user arguments (use_CPU, verbose_level, precision)
- Instantiates core AIBenchmark object
- Calls `.run()`, `.run_inference()`, or `.run_training()`
b. **Core Orchestration** (core.py)
- Loads configuration from config.py
- Iterates over selected tests/sections
- Delegates to Model Loader, Data Loader, and Execution Engine
c. **Configuration** (config.py)
d. **Data Management** (data/*)
- Provides sample inputs: images, text, segmentation maps
- Data loading util in utils.py
e. **Model Management** (models/*.meta + model_utils.py, update_utils.py)
- Stores TensorFlow meta-graph files for each pre-trained network
- model_utils builds TF Graph + Session and restores weights
- update_utils checks remote repo for new model versions
f. **Execution Engine** (TensorFlow runtime)
- Runs inference/training loops according to precision and iterations
- Reports timing and resource usage
g. **Results & Reporting** (utils.py + console)
- Aggregates timings, prints summary or detailed logs
- Returns Python dict of results from .run()
h. **Testing & CI** (tests/, .travis.yml, tox.ini)
- Unit tests validate console, utils, update logic
- Functional tests simulate full benchmark runs

##### Architectural Patterns & Design Principles
- **Modular separation**: CLI vs Core vs Data vs Models
- **Pluggable tests**: New tests or models can be added by dropping .meta files + config entries
- **Single entry-point API**: AIBenchmark class abstracts complexity
- **Dependency Injection** (to some extent): Pass flags to override CPU/GPU usage
- **Containerization support** via Dockerfiles

- Clear component names: “CLI (ai-benchmark)”, “AIBenchmark Core”, “Config Module”, “Data Loader”, “Model Loader”, “TensorFlow Engine”, “Hardware Layer”, “Results Reporter”.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        CLIScript["CLI Script"]:::python
        Console["Console Interface"]:::python
    end

    %% Core
    subgraph "Core Layer"
        Core["AIBenchmark Core"]:::python
    end

    %% Configuration
    subgraph "Configuration"
        Config["Config Module"]:::python
    end

    %% Data Management
    subgraph "Data Management"
        DataLoader["Data Loader"]:::python
        DataAssets["Sample Datasets"]:::data
    end

    %% Model Management
    subgraph "Model Management"
        ModelUtils["Model Utilities"]:::python
        UpdateUtils["Update Utilities"]:::python
        ModelFiles["Model Files (.meta)"]:::data
    end

    %% Execution Engine
    subgraph "Execution Engine"
        TFEngine["TensorFlow Runtime"]:::external
    end

    %% Hardware
    subgraph "Hardware Layer"
        Hardware["CPU / GPU / TPU"]:::external
    end

    %% Results & Reporting
    subgraph "Results & Reporting"
        ResultsFmt["Result Formatting"]:::python
        ConsoleOut["Console Output"]:::python
    end

    %% Testing & CI
    subgraph "Testing & CI"
        Tests["Unit & Functional Tests"]:::python
        CI["CI Config (.travis.yml, tox.ini)"]:::python
    end

    %% Packaging & Distribution
    subgraph "Packaging & Distribution"
        Packaging["setup.py, setup.cfg, MANIFEST.in"]:::python
    end

    %% Containerization
    subgraph "Containerization"
        Docker["Dockerfiles"]:::external
    end

    %% Connections
    CLIScript -->|"parses args"| Core
    Console -->|"invokes .run()"| Core
    Core -->|"loads config"| Config
    Core -->|"fetches sample data"| DataLoader
    DataLoader -->|"reads assets"| DataAssets
    Core -->|"builds & restores"| ModelUtils
    ModelUtils -->|"loads meta-graph"| ModelFiles
    Core -->|"checks updates"| UpdateUtils
    Core -->|"executes session"| TFEngine
    TFEngine -->|"runs on"| Hardware
    Core -->|"formats results"| ResultsFmt
    ResultsFmt -->|"print summary"| ConsoleOut
    Tests -->|"validates modules"| Core
    CI -->|"executes tests & builds"| Docker

    %% Click Events
    click CLIScript "https://github.com/cloudmercato/ai-benchmark/tree/main/bin/ai-benchmark"
    click Console "https://github.com/cloudmercato/ai-benchmark/blob/main/ai_benchmark/console.py"
    click Core "https://github.com/cloudmercato/ai-benchmark/blob/main/ai_benchmark/core.py"
    click Config "https://github.com/cloudmercato/ai-benchmark/blob/main/ai_benchmark/config.py"
    click DataLoader "https://github.com/cloudmercato/ai-benchmark/blob/main/ai_benchmark/utils.py"
    click DataAssets "https://github.com/cloudmercato/ai-benchmark/tree/main/ai_benchmark/data/"
    click ModelUtils "https://github.com/cloudmercato/ai-benchmark/blob/main/ai_benchmark/model_utils.py"
    click UpdateUtils "https://github.com/cloudmercato/ai-benchmark/blob/main/ai_benchmark/update_utils.py"
    click ModelFiles "https://github.com/cloudmercato/ai-benchmark/tree/main/ai_benchmark/models/"
    click ResultsFmt "https://github.com/cloudmercato/ai-benchmark/blob/main/ai_benchmark/utils.py"
    click ConsoleOut "https://github.com/cloudmercato/ai-benchmark/blob/main/ai_benchmark/console.py"
    click Tests "https://github.com/cloudmercato/ai-benchmark/tree/main/ai_benchmark/tests/"
    click CI "https://github.com/cloudmercato/ai-benchmark/blob/main/.travis.yml"
    click Packaging "https://github.com/cloudmercato/ai-benchmark/blob/main/setup.py"
    click Docker "https://github.com/cloudmercato/ai-benchmark/tree/main/Dockerfile"

    %% Styles
    classDef python fill:#D6E8FF,stroke:#1F77B4,color:#1F77B4;
    classDef data fill:#FFE1A8,stroke:#FF8C00,color:#FF8C00;
    classDef external fill:#D3FFD3,stroke:#2CA02C,color:#2CA02C;
```
