How mit-han-lab/once-for-all works
Project type and purpose
• This is a research‐oriented PyTorch library (“Once-for-All” or OFA) for training a single over-parameterized neural network and deriving optimized sub-networks for diverse deployment scenarios. It provides:
• Training scripts (distributed via Horovod/MPI)
• Model zoo of elastic networks and specialized networks
• NAS tools (accuracy & latency predictors, search algorithms)
• Data‐loading abstractions for ImageNet
• Utilities (flops counting, dynamic layers, etc.)
File structure
• Top‐level scripts:
• train_ofa_net.py – orchestrates distributed training of the supernet
• eval_ofa_net.py / eval_specialized_net.py – evaluate performance on ImageNet
• build.sh, setup.py, requirements.txt, hubconf.py – packaging, CI, publishing
• Core library under ofa/:
• ofa/imagenet_classification: data_providers, elastic_nn (dynamic modules, networks, training, utils), run_manager
• ofa/model_zoo.py: factory for OFA and specialized nets
• ofa/nas: accuracy_predictor, efficiency_predictor, search_algorithm
• ofa/utils: common_tools, flops_counter, my_dataloader, pytorch_utils, custom layers/modules
• tutorial folder: example Jupyter notebooks
README for insights
• Confirms usage patterns (sample_active_subnet, set_active_subnet, ofa_specialized)
• Shows distributed training commands (MPI / Horovod)
• Highlights core dependencies (PyTorch, Horovod, ImageNet)
• User Scripts (train_ofa_net.py, eval_*.py)
• Run Manager & Scheduler (distributed orchestrator using Horovod/MPI)
• Data Layer (ImageNet dataset + ofa/utils/my_dataloader + data_providers)
• Supernet & Elastic NN Modules (dynamic_layers.py, dynamic_op.py, ofa_mbv3, ofa_proxyless, ofa_resnets)
• Model Zoo (ofa/model_zoo.py)
• NAS Suite
– Accuracy Predictor (arch_encoder, acc_predictor)
– Efficiency Predictor (latency_lookup_table)
– Search Algorithms (evolution.py)
• Specialized Network Builder (networks derived via predictors + fine‐tuning)
• Utilities & Common Tools (flops_counter, pytorch_utils, common_tools)
• External Dependencies (PyTorch, Horovod/MPI, Python environment)
1. User invokes train_ofa_net.py → Run Manager launches distributed workers
2. Each worker loads Data Layer → feeds samples to Elastic NN Modules
3. Training Loop (progressive shrinking) in elastic_nn/training uses dynamic_layers & utils
4. Trained supernet registered in Model Zoo
5. For specialization: User runs ofa_specialized → NAS Suite (accuracy + latency predictors) + Search Algorithm → yields specialized architectures → fine‐tunes them using Run Manager
6. eval_ofa_net.py / eval_specialized_net.py load networks from Model Zoo → feed Data Layer → compute metrics
c. Architectural patterns & design principles:
• Modular separation: data, model definition, training, NAS, evaluation
• Plugin‐style model zoo for dynamic subnets
• Pipeline orchestration via Run Manager for distributed execution
• Predict-then-search NAS approach (predictors + evolutionary search)
• Reusable utility layer for PyTorch extension (custom layers, flops/latency estimation)
d. Key technologies & libraries:
• PyTorch (core DL framework)
• Horovod / MPI (distributed training)
• Python 3.6+
• ImageNet dataset
• numpy, tqdm (utilities)