How robertbabayann/benchmark_v2 works
OptimBench benchmarks PyTorch optimizers across machine-learning tasks. The main pipeline resolves a task, calibrates a reference optimizer to derive targets, tunes the requested optimizer at several trial budgets, runs seeded evaluations, and reports target-reaching steps and cost-adjusted evaluation counts. Separate workflows measure optimizer step cost, calibrate search spaces, run benchmark suites, and summarize results. Task implementations span vision, graph, language, physics-informed, and recommendation problems. The README is empty; relationships for unsampled workflows rely on indexed imports, and no unshown runtime details are assumed.
Main entry points: common.py Task and data resolution · pipeline.py Evaluation pipeline · registry.py Optimizer registry · presets.py Preset management
Components 28
Workflows
- Pipeline runner
scripts/run_pipeline.py - Benchmark runner
scripts/run_benchmark.py - Calibration runner
scripts/run_calibration.py - Cost runner
scripts/run_cost.py - Goal calibration
scripts/run_goal_calibration.py - Report runner
scripts/run_report.py - Task and data resolution
scripts/common.py - Dataset preparation
scripts/prepare_data.py - Workflow settings
scripts/settings.py
Benchmark engine
- Evaluation pipeline
src/optimbench/pipeline.py - Hyperparameter tuning
src/optimbench/tuning.py - Search calibration
src/optimbench/calibration.py - Optimizer registry
src/optimbench/registry.py - Synthetic test functions
src/optimbench/testfunctions.py - Target calibration
src/optimbench/targets.py - Evaluation metrics
src/optimbench/metrics.py - Step-cost measurement
src/optimbench/cost.py - Device utilities
src/optimbench/device.py
Task workloads
- Task execution
src/optimbench/tasks/__init__.py - Vision tasks
src/optimbench/tasks/cv.py - Graph task
src/optimbench/tasks/gnn.py - Language tasks
src/optimbench/tasks/nlp.py - Physics task
src/optimbench/tasks/pinn.py - Recommendation task
src/optimbench/tasks/recsys.py
Presets and results
- Preset management
src/optimbench/presets.py - Search-space preset
src/optimbench/presets/search_space.json - Cost preset
src/optimbench/presets/cost.json
Other components
- Benchmark user
Connections 40
- Benchmark user → Pipeline runner: starts pipeline
scripts/run_pipeline.py - Pipeline runner → Task and data resolution: resolves task
scripts/run_pipeline.py - Pipeline runner → Device utilities: selects device
scripts/run_pipeline.py - Pipeline runner → Evaluation pipeline: runs stages
scripts/run_pipeline.py - Pipeline runner → Workflow settings: reads settings
scripts/settings.py - Benchmark runner → Evaluation pipeline: runs benchmarks
scripts/run_benchmark.py - Benchmark runner → Task and data resolution: resolves tasks
scripts/run_benchmark.py - Benchmark runner → Device utilities: selects device
scripts/run_benchmark.py - Calibration runner → Search calibration: runs calibration
scripts/run_calibration.py - Calibration runner → Preset management: loads and saves
scripts/run_calibration.py - Calibration runner → Optimizer registry: selects optimizers
scripts/run_calibration.py - Cost runner → Step-cost measurement: measures step time
scripts/run_cost.py - Cost runner → Task and data resolution: resolves tasks
scripts/run_cost.py - Cost runner → Optimizer registry: gets optimizer specs
scripts/run_cost.py - Goal calibration → Evaluation pipeline: runs reference
scripts/run_goal_calibration.py - Goal calibration → Task and data resolution: resolves task
scripts/run_goal_calibration.py - Report runner → Evaluation metrics: uses metrics
scripts/run_report.py - Task and data resolution → Optimizer registry: lists optimizers
scripts/common.py - Task and data resolution → Dataset preparation: prepares datasets
scripts/common.py - Task and data resolution → Vision tasks: resolves classes
scripts/common.py - Task and data resolution → Graph task: resolves classes
scripts/common.py - Task and data resolution → Language tasks: resolves classes
scripts/common.py - Task and data resolution → Physics task: resolves classes
scripts/common.py - Task and data resolution → Recommendation task: resolves classes
scripts/common.py - Task execution → Optimizer registry: builds optimizer
src/optimbench/tasks/__init__.py - Task execution → Device utilities: moves batches
src/optimbench/tasks/__init__.py - Evaluation pipeline → Task execution: runs task
src/optimbench/pipeline.py - Evaluation pipeline → Optimizer registry: gets parameter specs
src/optimbench/pipeline.py - Evaluation pipeline → Preset management: loads calibration and cost
src/optimbench/pipeline.py - Evaluation pipeline → Hyperparameter tuning: tunes optimizer
src/optimbench/pipeline.py - Evaluation pipeline → Target calibration: calibrates targets
src/optimbench/pipeline.py - Evaluation pipeline → Evaluation metrics: aggregates outcomes
src/optimbench/pipeline.py - Evaluation pipeline → Step-cost measurement: measures fallback cost
src/optimbench/pipeline.py - Search calibration → Optimizer registry: builds optimizer
src/optimbench/calibration.py - Search calibration → Synthetic test functions: evaluates functions
src/optimbench/calibration.py - Preset management → Search calibration: computes missing preset
src/optimbench/presets.py - Preset management → Optimizer registry: records optimizer source
src/optimbench/presets.py - Preset management → Search-space preset: reads and writes
src/optimbench/presets.py - Preset management → Cost preset: reads cost data
src/optimbench/presets.py - Step-cost measurement → Optimizer registry: builds optimizer
src/optimbench/cost.py