How aidinhamedi/optimizer-benchmark works
Project Type & Purpose
• This is an open-source benchmarking suite (Python CLI tool) for evaluating and comparing PyTorch optimizers on 2D test functions, producing both raw results and a static website of rankings and trajectory visualizations.
High-Level Architecture Components
a. CLI Orchestrator
• runner.py: entry point that reads config.toml, kicks off benchmark runs.
• Makefile / uv (uv.sync): wrapper for installation & common commands.
b. Benchmark Engine
• benchmark/criterion.py: defines evaluation criteria.
• benchmark/evaluate.py: runs the optimization loops.
• benchmark/functions/: suite of 2D test functions.
• benchmark/utils/ (executor.py, model.py, surface.py): support code for running optimization, representing surfaces, logging.
c. Hyperparameter Tuning
• Optuna integration inside evaluate.py (search loops).
d. Visualization Pipeline
• benchmark/visualizer.py: generates trajectory plots.
• tools/doc_visualizations.py & tools/gen_comparison.py: produces the HTML snippets & summary tables for the docs site using Jinja templates.
e. Result Storage
• results/results.json: aggregated benchmark data.
• PNG/HTML plots under docs/vis or results subfolders.
f. Documentation / Website
• docs/: static website (HTML, CSS, JS) generated via Jinja templates, updated from tools/.
• GitHub Pages deployment (site lives at aidinhamedi.github.io/Optimizer-Benchmark).
g. External Dependencies
• PyTorch, pytorch_optimizer library, Optuna, Jinja2, toml parser, any plotting libs (e.g., Matplotlib or Plotly).
Relationships & Data Flow
1. User invokes python runner.py (or make benchmark)
2. Runner loads config.toml →
3. For each optimizer & function:
• evaluate.py uses functions/ to compute gradients,
• Optuna for hyperparameter search,
• Records metrics via criterion.py →
• Writes raw data to results/results.json.
4. benchmark/visualizer.py reads results.json → generates per-optimizer & per-function trajectory plots (HTML/PNG) in docs/vis.
5. tools/ scripts read results.json + plots → render summary pages via Jinja templates → place final HTML, CSS, JS under docs/.
6. docs/ becomes the GitHub Pages site.
Architectural Patterns & Principles
• Modular • Separation of concerns: CLI, benchmarking, tuning, visualization, documentation.
• Pipeline: data flows from raw runs → results → visualizations → static site.
• Extensibility: easily plug in new functions or optimizers by adding files under benchmark/functions or adjusting config.toml.
• Configuration-driven: all parameters in config.toml.