# robertbabayann/benchmark_v2 architecture

> Architecture diagram of the GitHub repository robertbabayann/benchmark_v2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/robertbabayann/benchmark_v2
- Repository: https://github.com/robertbabayann/benchmark_v2

## Overview

OptimBench benchmarks PyTorch optimizers across machine-learning tasks. The main pipeline resolves a task, calibrates a reference optimizer to derive targets, tunes the requested optimizer at several trial budgets, runs seeded evaluations, and reports target-reaching steps and cost-adjusted evaluation counts. Separate workflows measure optimizer step cost, calibrate search spaces, run benchmark suites, and summarize results. Task implementations span vision, graph, language, physics-informed, and recommendation problems. The README is empty; relationships for unsampled workflows rely on indexed imports, and no unshown runtime details are assumed.

Main entry points: [`scripts/common.py`](https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/common.py) (Task and data resolution), [`src/optimbench/pipeline.py`](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/pipeline.py) (Evaluation pipeline), [`src/optimbench/registry.py`](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/registry.py) (Optimizer registry), [`src/optimbench/presets.py`](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/presets.py) (Preset management)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflows["Workflows"]
  node_pipeline_cli["Pipeline runner<br/>[run_pipeline.py]"]
  node_benchmark_cli["Benchmark runner<br/>[run_benchmark.py]"]
  node_calibration_cli["Calibration runner<br/>[run_calibration.py]"]
  node_cost_cli["Cost runner<br/>[run_cost.py]"]
  node_goal_cli["Goal calibration"]
  node_report_cli["Report runner<br/>[run_report.py]"]
  node_task_resolution["Task and data resolution<br/>[common.py]"]
  node_data_prep["Dataset preparation<br/>[prepare_data.py]"]
  node_settings["Workflow settings<br/>[settings.py]"]
end

subgraph group_engine["Benchmark engine"]
  node_pipeline["Evaluation pipeline<br/>[pipeline.py]"]
  node_tuning["Hyperparameter tuning<br/>[tuning.py]"]
  node_calibration["Search calibration<br/>[calibration.py]"]
  node_optimizer_registry["Optimizer registry<br/>[registry.py]"]
  node_test_functions["Synthetic test functions<br/>[testfunctions.py]"]
  node_targets["Target calibration<br/>[targets.py]"]
  node_metrics["Evaluation metrics<br/>[metrics.py]"]
  node_cost["Step-cost measurement<br/>[cost.py]"]
  node_device["Device utilities<br/>[device.py]"]
end

subgraph group_tasks["Task workloads"]
  node_task_base["Task execution<br/>[__init__.py]"]
  node_cv_tasks["Vision tasks<br/>[cv.py]"]
  node_gnn_tasks["Graph task<br/>[gnn.py]"]
  node_nlp_tasks["Language tasks<br/>[nlp.py]"]
  node_pinn_tasks["Physics task<br/>[pinn.py]"]
  node_recsys_tasks["Recommendation task<br/>[recsys.py]"]
end

subgraph group_presets["Presets and results"]
  node_presets["Preset management<br/>[presets.py]"]
  node_search_preset["Search-space preset<br/>[search_space.json]"]
  node_cost_preset["Cost preset<br/>[cost.json]"]
end

node_user(("Benchmark user"))

node_user -->|"starts pipeline"| node_pipeline_cli
node_pipeline_cli -->|"resolves task"| node_task_resolution
node_pipeline_cli -->|"selects device"| node_device
node_pipeline_cli -->|"runs stages"| node_pipeline
node_pipeline_cli -.->|"reads settings"| node_settings
node_benchmark_cli -->|"runs benchmarks"| node_pipeline
node_benchmark_cli -->|"resolves tasks"| node_task_resolution
node_benchmark_cli -->|"selects device"| node_device
node_calibration_cli -->|"runs calibration"| node_calibration
node_calibration_cli -->|"loads and saves"| node_presets
node_calibration_cli -->|"selects optimizers"| node_optimizer_registry
node_cost_cli -->|"measures step time"| node_cost
node_cost_cli -->|"resolves tasks"| node_task_resolution
node_cost_cli -->|"gets optimizer specs"| node_optimizer_registry
node_goal_cli -->|"runs reference"| node_pipeline
node_goal_cli -->|"resolves task"| node_task_resolution
node_report_cli -->|"uses metrics"| node_metrics
node_task_resolution -->|"lists optimizers"| node_optimizer_registry
node_task_resolution -->|"prepares datasets"| node_data_prep
node_task_resolution -->|"resolves classes"| node_cv_tasks
node_task_resolution -->|"resolves classes"| node_gnn_tasks
node_task_resolution -->|"resolves classes"| node_nlp_tasks
node_task_resolution -->|"resolves classes"| node_pinn_tasks
node_task_resolution -->|"resolves classes"| node_recsys_tasks
node_task_base -->|"builds optimizer"| node_optimizer_registry
node_task_base -->|"moves batches"| node_device
node_pipeline -->|"runs task"| node_task_base
node_pipeline -->|"gets parameter specs"| node_optimizer_registry
node_pipeline -->|"loads calibration and cost"| node_presets
node_pipeline -->|"tunes optimizer"| node_tuning
node_pipeline -->|"calibrates targets"| node_targets
node_pipeline -->|"aggregates outcomes"| node_metrics
node_pipeline -->|"measures fallback cost"| node_cost
node_calibration -->|"builds optimizer"| node_optimizer_registry
node_calibration -->|"evaluates functions"| node_test_functions
node_presets -->|"computes missing preset"| node_calibration
node_presets -->|"records optimizer source"| node_optimizer_registry
node_presets -->|"reads and writes"| node_search_preset
node_presets -->|"reads cost data"| node_cost_preset
node_cost -->|"builds optimizer"| node_optimizer_registry

click node_pipeline_cli "https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/run_pipeline.py"
click node_benchmark_cli "https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/run_benchmark.py"
click node_calibration_cli "https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/run_calibration.py"
click node_cost_cli "https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/run_cost.py"
click node_goal_cli "https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/run_goal_calibration.py"
click node_report_cli "https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/run_report.py"
click node_task_resolution "https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/common.py"
click node_data_prep "https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/prepare_data.py"
click node_settings "https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/settings.py"
click node_pipeline "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/pipeline.py"
click node_tuning "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tuning.py"
click node_calibration "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/calibration.py"
click node_optimizer_registry "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/registry.py"
click node_test_functions "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/testfunctions.py"
click node_targets "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/targets.py"
click node_metrics "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/metrics.py"
click node_cost "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/cost.py"
click node_device "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/device.py"
click node_task_base "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tasks/__init__.py"
click node_cv_tasks "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tasks/cv.py"
click node_gnn_tasks "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tasks/gnn.py"
click node_nlp_tasks "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tasks/nlp.py"
click node_pinn_tasks "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tasks/pinn.py"
click node_recsys_tasks "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tasks/recsys.py"
click node_presets "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/presets.py"
click node_search_preset "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/presets/search_space.json"
click node_cost_preset "https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/presets/cost.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline_cli,node_benchmark_cli,node_calibration_cli,node_cost_cli,node_goal_cli,node_report_cli,node_task_resolution,node_data_prep,node_settings,node_user toneBlue
class node_pipeline,node_tuning,node_calibration,node_optimizer_registry,node_test_functions,node_targets,node_metrics,node_cost,node_device toneAmber
class node_task_base,node_cv_tasks,node_gnn_tasks,node_nlp_tasks,node_pinn_tasks,node_recsys_tasks toneMint
class node_presets,node_search_preset,node_cost_preset toneRose
```

## Components

### Workflows

- [Pipeline runner](https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/run_pipeline.py): `scripts/run_pipeline.py`
- [Benchmark runner](https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/run_benchmark.py): `scripts/run_benchmark.py`
- [Calibration runner](https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/run_calibration.py): `scripts/run_calibration.py`
- [Cost runner](https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/run_cost.py): `scripts/run_cost.py`
- [Goal calibration](https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/run_goal_calibration.py): `scripts/run_goal_calibration.py`
- [Report runner](https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/run_report.py): `scripts/run_report.py`
- [Task and data resolution](https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/common.py): `scripts/common.py`
- [Dataset preparation](https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/prepare_data.py): `scripts/prepare_data.py`
- [Workflow settings](https://github.com/robertbabayann/benchmark_v2/blob/main/scripts/settings.py): `scripts/settings.py`

### Benchmark engine

- [Evaluation pipeline](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/pipeline.py): `src/optimbench/pipeline.py`
- [Hyperparameter tuning](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tuning.py): `src/optimbench/tuning.py`
- [Search calibration](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/calibration.py): `src/optimbench/calibration.py`
- [Optimizer registry](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/registry.py): `src/optimbench/registry.py`
- [Synthetic test functions](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/testfunctions.py): `src/optimbench/testfunctions.py`
- [Target calibration](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/targets.py): `src/optimbench/targets.py`
- [Evaluation metrics](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/metrics.py): `src/optimbench/metrics.py`
- [Step-cost measurement](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/cost.py): `src/optimbench/cost.py`
- [Device utilities](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/device.py): `src/optimbench/device.py`

### Task workloads

- [Task execution](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tasks/__init__.py): `src/optimbench/tasks/__init__.py`
- [Vision tasks](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tasks/cv.py): `src/optimbench/tasks/cv.py`
- [Graph task](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tasks/gnn.py): `src/optimbench/tasks/gnn.py`
- [Language tasks](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tasks/nlp.py): `src/optimbench/tasks/nlp.py`
- [Physics task](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tasks/pinn.py): `src/optimbench/tasks/pinn.py`
- [Recommendation task](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/tasks/recsys.py): `src/optimbench/tasks/recsys.py`

### Presets and results

- [Preset management](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/presets.py): `src/optimbench/presets.py`
- [Search-space preset](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/presets/search_space.json): `src/optimbench/presets/search_space.json`
- [Cost preset](https://github.com/robertbabayann/benchmark_v2/blob/main/src/optimbench/presets/cost.json): `src/optimbench/presets/cost.json`

### Other components

- **Benchmark user**

## Connections

- Benchmark user → Pipeline runner: starts pipeline (evidence: `scripts/run_pipeline.py`)
- Pipeline runner → Task and data resolution: resolves task (evidence: `scripts/run_pipeline.py`)
- Pipeline runner → Device utilities: selects device (evidence: `scripts/run_pipeline.py`)
- Pipeline runner → Evaluation pipeline: runs stages (evidence: `scripts/run_pipeline.py`)
- Pipeline runner → Workflow settings: reads settings (evidence: `scripts/settings.py`)
- Benchmark runner → Evaluation pipeline: runs benchmarks (evidence: `scripts/run_benchmark.py`)
- Benchmark runner → Task and data resolution: resolves tasks (evidence: `scripts/run_benchmark.py`)
- Benchmark runner → Device utilities: selects device (evidence: `scripts/run_benchmark.py`)
- Calibration runner → Search calibration: runs calibration (evidence: `scripts/run_calibration.py`)
- Calibration runner → Preset management: loads and saves (evidence: `scripts/run_calibration.py`)
- Calibration runner → Optimizer registry: selects optimizers (evidence: `scripts/run_calibration.py`)
- Cost runner → Step-cost measurement: measures step time (evidence: `scripts/run_cost.py`)
- Cost runner → Task and data resolution: resolves tasks (evidence: `scripts/run_cost.py`)
- Cost runner → Optimizer registry: gets optimizer specs (evidence: `scripts/run_cost.py`)
- Goal calibration → Evaluation pipeline: runs reference (evidence: `scripts/run_goal_calibration.py`)
- Goal calibration → Task and data resolution: resolves task (evidence: `scripts/run_goal_calibration.py`)
- Report runner → Evaluation metrics: uses metrics (evidence: `scripts/run_report.py`)
- Task and data resolution → Optimizer registry: lists optimizers (evidence: `scripts/common.py`)
- Task and data resolution → Dataset preparation: prepares datasets (evidence: `scripts/common.py`)
- Task and data resolution → Vision tasks: resolves classes (evidence: `scripts/common.py`)
- Task and data resolution → Graph task: resolves classes (evidence: `scripts/common.py`)
- Task and data resolution → Language tasks: resolves classes (evidence: `scripts/common.py`)
- Task and data resolution → Physics task: resolves classes (evidence: `scripts/common.py`)
- Task and data resolution → Recommendation task: resolves classes (evidence: `scripts/common.py`)
- Task execution → Optimizer registry: builds optimizer (evidence: `src/optimbench/tasks/__init__.py`)
- Task execution → Device utilities: moves batches (evidence: `src/optimbench/tasks/__init__.py`)
- Evaluation pipeline → Task execution: runs task (evidence: `src/optimbench/pipeline.py`)
- Evaluation pipeline → Optimizer registry: gets parameter specs (evidence: `src/optimbench/pipeline.py`)
- Evaluation pipeline → Preset management: loads calibration and cost (evidence: `src/optimbench/pipeline.py`)
- Evaluation pipeline → Hyperparameter tuning: tunes optimizer (evidence: `src/optimbench/pipeline.py`)
- Evaluation pipeline → Target calibration: calibrates targets (evidence: `src/optimbench/pipeline.py`)
- Evaluation pipeline → Evaluation metrics: aggregates outcomes (evidence: `src/optimbench/pipeline.py`)
- Evaluation pipeline → Step-cost measurement: measures fallback cost (evidence: `src/optimbench/pipeline.py`)
- Search calibration → Optimizer registry: builds optimizer (evidence: `src/optimbench/calibration.py`)
- Search calibration → Synthetic test functions: evaluates functions (evidence: `src/optimbench/calibration.py`)
- Preset management → Search calibration: computes missing preset (evidence: `src/optimbench/presets.py`)
- Preset management → Optimizer registry: records optimizer source (evidence: `src/optimbench/presets.py`)
- Preset management → Search-space preset: reads and writes (evidence: `src/optimbench/presets.py`)
- Preset management → Cost preset: reads cost data (evidence: `src/optimbench/presets.py`)
- Step-cost measurement → Optimizer registry: builds optimizer (evidence: `src/optimbench/cost.py`)
