How cloudmercato/ai-benchmark works
Project Type and Purpose
• This is an open-source Python library + CLI tool for measuring inference/training performance of deep-learning models on various hardware (CPUs, GPUs, TPUs).
• Key technologies: Python, TensorFlow 1.x/2.x, Docker, CUDA/cuDNN.
File Structure
Top-level components:
• bin/ai-benchmark – CLI entry-point script
• ai_benchmark/ – core Python package
– config.py – definitions of tests, sections, default parameters
– console.py – command-line interface wrapper around the core
– core.py – AIBenchmark class orchestrating test runs
– data/ – bundled small sample datasets for classification, segmentation, NLP, etc.
– model_utils.py – loading TensorFlow meta-graphs, building sessions
– models/ .meta files – pre-exported TensorFlow graph definitions
– update_utils.py – functionality to download or update model files
– utils.py – helper functions (logging, result formatting)
– tests/ – unit and functional tests for each module
• Dockerfile* – container definitions for CPU, GPU, PPC64LE builds
• .travis.yml, tox.ini – CI configurations
• setup.py, setup.cfg, MANIFEST.in – packaging & distribution
Readme Insights
• 42 individual tests across 19 sections
• Exposes high-level API (AIBenchmark.run(), .run_inference(), .run_training())
• CLI interface via ai-benchmark console command
• Supports advanced settings: CPU vs GPU, verbosity, precision
Main Components & Their Interactions
a. CLI Layer (bin/ai-benchmark + console.py)
• Parses user arguments (use_CPU, verbose_level, precision)
• Instantiates core AIBenchmark object
• Calls .run(), .run_inference(), or .run_training()
b. Core Orchestration (core.py)
• Loads configuration from config.py
• Iterates over selected tests/sections
• Delegates to Model Loader, Data Loader, and Execution Engine
c. Configuration (config.py)
d. Data Management (data/*)
• Provides sample inputs: images, text, segmentation maps
• Data loading util in utils.py
e. Model Management (models/*.meta + model_utils.py, update_utils.py)
• Stores TensorFlow meta-graph files for each pre-trained network
• model_utils builds TF Graph + Session and restores weights
• update_utils checks remote repo for new model versions
f. Execution Engine (TensorFlow runtime)
• Runs inference/training loops according to precision and iterations
• Reports timing and resource usage
g. Results & Reporting (utils.py + console)
• Aggregates timings, prints summary or detailed logs
• Returns Python dict of results from .run()
h. Testing & CI (tests/, .travis.yml, tox.ini)
• Unit tests validate console, utils, update logic
• Functional tests simulate full benchmark runs
Architectural Patterns & Design Principles
• Modular separation: CLI vs Core vs Data vs Models
• Pluggable tests: New tests or models can be added by dropping .meta files + config entries
• Single entry-point API: AIBenchmark class abstracts complexity
• Dependency Injection (to some extent): Pass flags to override CPU/GPU usage
• Containerization support via Dockerfiles
• Clear component names: “CLI (ai-benchmark)”, “AIBenchmark Core”, “Config Module”, “Data Loader”, “Model Loader”, “TensorFlow Engine”, “Hardware Layer”, “Results Reporter”.