How juntang-zhuang/adabelief-optimizer works
Project Type and Purpose
• This is an open-source research code repository centered on the AdaBelief optimizer.
• It provides:
• The core AdaBelief implementations for PyTorch and TensorFlow (packaged under pypi_packages).
• A suite of experiments (under PyTorch_Experiments and TensorFlow_Experiments) demonstrating AdaBelief vs. other optimizers on tasks like LSTM language modeling, CIFAR10/ImageNet classification, GAN training, and more.
• Jupyter notebooks and scripts for visualization and reproduction of results.
File Structure Analysis
• Top-level:
LICENSE, README.md, imgs/ (figures), pypi_packages/ (versioned optimizer source), PyTorch_Experiments/, TensorFlow_Experiments/
• pypi_packages/
• adabelief_pytorchX.Y.Z/ – core AdaBelief code (AdaBelief.py, setup.py, etc.)
• adabelief_tfX.Y.Z/ – TensorFlow counterpart
• ranger_adabelief – combined optimizer variant
• PyTorch_Experiments/
• LSTM/ – data loaders, model.py, main.py, optimizer variants, curve output
• classification_cifar10/ – models/, optimizers/, run.sh, visualization.ipynb, curve data
• imagenet/, wgan/, wgan-gp/ – each with main scripts, optimizers folder, run.sh, readme.md
• TensorFlow_Experiments/
• convolutional_network.ipynb
• imgs/ – static assets used in README and notebooks
README Insights
• Describes the AdaBelief algorithm, hyper-parameters, update rules, and comparisons.
• Directs users to examples in PyTorch_Experiments (via run.sh) and the pypi_packages for the “official” source.
a. Main Components (group into logical tiers or modules):
1. Core Library
• AdaBelief-PyTorch (pypi_packages/adabelief_pytorchX.Y.Z)
• AdaBelief-TF (pypi_packages/adabelief_tfX.Y.Z)
• RangerAdaBelief (pypi_packages/ranger_adabelief)
2. Experiment Framework
• Data Sources (PTB, CIFAR10, ImageNet, GAN toy datasets)
• Data Loaders / Preprocessors (data.py, dataloader.py, getdata.sh)
• Model Definitions (LSTM, ResNet, DenseNet, VGG, SENet, ShuffleNet, Xception, GAN generator/discriminator)
• Optimizer Modules (wrap AdaBelief and competitor optimizers: AdamW, RAdam, Yogi, MSVAG, Fromage, SGD)
• Training Orchestrators
• main.py / run.sh scripts
• experiment-specific entry points (run_all_layerX.py, run.sh)
• Output Stores
• curve/ directories (CSV or serialized metrics)
• checkpoint exports
3. Visualization & Reporting
• Jupyter notebooks (LSTM_test.ipynb, visualization.ipynb, convolutional_network.ipynb)
• imgs/ folder for embedding charts in README
4. Packaging & Distribution
• setup.py in pypi_packages modules
• Versioned folders for release management
• README.md at root describing installation, usage, and citations
b. Relationships & Data Flow
1. Data Source → Data Loader → Model Definition
2. Model + AdaBelief Optimizer (or other optimizer) → Training Loop
3. Training Loop → Metric Collection → ‘curve’ Output Files
4. curve Files → Visualization Notebooks → Plots in imgs/ and notebooks
5. Core Library Modules → Imported by Experiment Framework
6. Packaging Modules → pip install → External Users
c. Architectural Patterns & Principles
• Modular separation: Core library is decoupled from experiment code.
• Versioned packaging: each release lives in its own pypi_packages subfolder.
• Reproducible experiments: standardized run.sh / scripts for each task.
• Extensibility: support for adding new optimizers, models, datasets.
d. Key Technologies
• PyTorch (training, model definition)
• TensorFlow (in TensorFlow_Experiments and adabelief_tf)
• Python scripting (main.py, run.sh)
• Jupyter notebooks for reporting
• pip packaging with setup.py