How aidinhamedi/ml-optimizer-benchmark works
Project type and purpose
• This is a command-line benchmarking suite (a Python “tool”) for evaluating PyTorch optimizers on 2D test functions.
• It loads a TOML config (config.toml), runs hyperparameter tuning with Optuna, evaluates optimizer behavior on various functions in benchmark/functions, then visualizes the trajectories and writes results to JSON/plots.
File structure
• Entry point: runner.py
• Core package: benchmark/
• hypertune.py – orchestrates Optuna searches
• evaluate.py – evaluates one optimizer on one function
• functions/ – definitions of 2D test functions (e.g. ackley.py, rastrigin.py, etc.)
• utils/ – helpers:
– executor.py – runs the optimization loop on a given function + model
– model.py – wraps PyTorch modules & optimizer from pytorch_optimizer
– surface.py – generates grid data for contour plots
• Visualization: benchmark/visualizer.py
• Config: config.toml
• Results folder: results/ (JSON + images)
• Docs and tools/ for generating/ syncing documentation
Extract call-flow from README & filenames
• runner.py loads config → calls the hyperparameter‐tuning driver → calls evaluation → calls visualization → writes to results/.
Main components
A. Runner (CLI)
• Function: main() in runner.py
• Calls:
– ConfigLoader.load(config.toml)
– BenchmarkTuner.run_all() (in benchmark/hypertune.py)
– Visualizer.generate_all() (in benchmark/visualizer.py)
B. BenchmarkTuner (in benchmark/hypertune.py)
• Method: run_all() iterates optimizers & functions
• Calls per‐optimizer: run_hyperparameter_search(optimizer_name)
• Under the hood uses Optuna’s study.optimize() which calls objective(trial)
C. Evaluator (in benchmark/evaluate.py)
• Function: evaluate_one(optimizer, function, hyperparams)
• Calls:
– Executor.execute(model, func, iterations)
D. Executor (in benchmark/utils/executor.py)
• Class: Executor
• Method: execute()
– Instantiates Model (from model.py) with a PyTorch network + chosen optimizer
– Loops: compute gradient on func(point) → update parameters → record trajectory
E. Model (in benchmark/utils/model.py)
• Wraps PyTorch module + pytorch_optimizer call
F. Surface (in benchmark/utils/surface.py)
• Generates 2D grid of loss values for contour plots
G. Visualizer (in benchmark/visualizer.py)
• Function: generate_plots(results.json, trajectories)
• Renders contour + overlayed optimizer paths → saves PNG/HTML under results/vis/
H. Outputs
• results/results.json
• results/vis/<optimizer>.png or .html
Relationships & interactions
• Data flow:
config.toml → Runner → BenchmarkTuner → Optuna trials → Evaluator → Executor → Model + Function → recorded trajectories → Visualizer → results/
Runner → BenchmarkTuner → Evaluator → Executor → Model & Function
Runner → Visualizer
• External services:
– PyTorch + pytorch_optimizer
– Optuna
Key architectural patterns & principles
• Modular layering:
– CLI layer (runner.py)
– Orchestration layer (hypertune.py)
– Evaluation layer (evaluate.py + utils)
– Visualization layer (visualizer.py)
• Separation of concerns: config, benchmarking logic, function definitions, plotting
• Plugin-style function repository: any .py in benchmark/functions/ is automatically picked up