# mit-han-lab/once-for-all architecture

> Architecture diagram of the GitHub repository mit-han-lab/once-for-all, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mit-han-lab/once-for-all
- Repository: https://github.com/mit-han-lab/once-for-all

## Overview

##### Project type and purpose
- This is a research‐oriented PyTorch library (“Once-for-All” or OFA) for training a single over-parameterized neural network and deriving optimized sub-networks for diverse deployment scenarios. It provides:
- Training scripts (distributed via Horovod/MPI)
- Model zoo of elastic networks and specialized networks
- NAS tools (accuracy & latency predictors, search algorithms)
- Data‐loading abstractions for ImageNet
- Utilities (flops counting, dynamic layers, etc.)

##### File structure
- Top‐level scripts:
- train_ofa_net.py – orchestrates distributed training of the supernet
- eval_ofa_net.py / eval_specialized_net.py – evaluate performance on ImageNet
- build.sh, setup.py, requirements.txt, hubconf.py – packaging, CI, publishing
- Core library under ofa/:
- ofa/imagenet_classification: data_providers, elastic_nn (dynamic modules, networks, training, utils), run_manager
- ofa/model_zoo.py: factory for OFA and specialized nets
- ofa/nas: accuracy_predictor, efficiency_predictor, search_algorithm
- ofa/utils: common_tools, flops_counter, my_dataloader, pytorch_utils, custom layers/modules
- tutorial folder: example Jupyter notebooks

##### README for insights
- Confirms usage patterns (sample_active_subnet, set_active_subnet, ofa_specialized)
- Shows distributed training commands (MPI / Horovod)
- Highlights core dependencies (PyTorch, Horovod, ImageNet)

- User Scripts (train_ofa_net.py, eval_*.py)
- Run Manager & Scheduler (distributed orchestrator using Horovod/MPI)
- Data Layer (ImageNet dataset + ofa/utils/my_dataloader + data_providers)
- Supernet & Elastic NN Modules (dynamic_layers.py, dynamic_op.py, ofa_mbv3, ofa_proxyless, ofa_resnets)
- Model Zoo (ofa/model_zoo.py)
- NAS Suite
– Accuracy Predictor (arch_encoder, acc_predictor)
– Efficiency Predictor (latency_lookup_table)
– Search Algorithms (evolution.py)
- Specialized Network Builder (networks derived via predictors + fine‐tuning)
- Utilities & Common Tools (flops_counter, pytorch_utils, common_tools)
- External Dependencies (PyTorch, Horovod/MPI, Python environment)

1. User invokes train_ofa_net.py → Run Manager launches distributed workers
2. Each worker loads Data Layer → feeds samples to Elastic NN Modules
3. Training Loop (progressive shrinking) in elastic_nn/training uses dynamic_layers & utils
4. Trained supernet registered in Model Zoo
5. For specialization: User runs ofa_specialized → NAS Suite (accuracy + latency predictors) + Search Algorithm → yields specialized architectures → fine‐tunes them using Run Manager
6. eval_ofa_net.py / eval_specialized_net.py load networks from Model Zoo → feed Data Layer → compute metrics

c. Architectural patterns & design principles:
- Modular separation: data, model definition, training, NAS, evaluation
- Plugin‐style model zoo for dynamic subnets
- Pipeline orchestration via Run Manager for distributed execution
- Predict-then-search NAS approach (predictors + evolutionary search)
- Reusable utility layer for PyTorch extension (custom layers, flops/latency estimation)

d. Key technologies & libraries:
- PyTorch (core DL framework)
- Horovod / MPI (distributed training)
- Python 3.6+
- ImageNet dataset
- numpy, tqdm (utilities)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Scripts (Blue)
    subgraph "User Scripts" 
        direction TB
        Train["train_ofa_net.py"]:::scripts
        EvalOFA["eval_ofa_net.py"]:::scripts
        EvalSpec["eval_specialized_net.py"]:::scripts
    end

    %% Run Manager & Scheduler (Green)
    subgraph "Run Manager & Scheduler"
        direction TB
        DistRunMgr["distributed_run_manager.py"]:::core
        RunMgr["run_manager.py"]:::core
        RunConfig["run_config.py"]:::core
    end

    %% Data Layer (Yellow)
    subgraph "Data Layer"
        direction TB
        IMProv["imagenet.py"]:::data
        BaseProv["base_provider.py"]:::data
        DLLoader["my_data_loader.py"]:::data
        DLWorker["my_data_worker.py"]:::data
        DistSampler["my_distributed_sampler.py"]:::data
        RandCrop["my_random_resize_crop.py"]:::data
    end

    %% Supernet & Elastic NN Modules (Green)
    subgraph "Supernet & Elastic NN Modules"
        direction TB
        DynLayers["dynamic_layers.py"]:::core
        DynOp["dynamic_op.py"]:::core
        MBV3["ofa_mbv3.py"]:::core
        ProxyNAS["ofa_proxyless.py"]:::core
        ResNets["ofa_resnets.py"]:::core
        ProgShrink["progressive_shrinking.py"]:::core
        ENNUtils["elastic_nn utils"]:::core
    end

    %% Model Zoo (Green)
    ModelZoo["model_zoo.py"]:::core

    %% NAS Suite (Green)
    subgraph "NAS Suite"
        direction TB
        subgraph "Accuracy Predictor"
            AccDS["acc_dataset.py"]:::core
            AccPred["acc_predictor.py"]:::core
            ArchEnc["arch_encoder.py"]:::core
        end
        LatTable["latency_lookup_table.py"]:::core
        EvolSearch["evolution.py"]:::core
    end

    %% Utilities & Common Tools (Green)
    subgraph "Utilities & Common Tools"
        direction TB
        CommonTools["common_tools.py"]:::core
        FlopsCnt["flops_counter.py"]:::core
        Layers["layers.py"]:::core
        PTMods["pytorch_modules.py"]:::core
        PTUtils["pytorch_utils.py"]:::core
    end

    %% External Dependencies (Orange)
    subgraph "External Dependencies"
        direction TB
        PyTorch["PyTorch"]:::external
        Horovod["Horovod/MPI"]:::external
        ImageNetDS["ImageNet Dataset"]:::external
        PythonEnv["Python3 / numpy / tqdm"]:::external
    end

    %% Connections
    Train -->|"MPI launch"| DistRunMgr
    EvalOFA -->|load supernet| ModelZoo
    EvalSpec -->|load specialized| ModelZoo

    DistRunMgr --> RunMgr
    RunMgr --> RunConfig

    RunMgr -->|"load data"| IMProv
    RunMgr -->|"load data"| BaseProv
    RunMgr --> DLLoader
    RunMgr --> DLWorker
    RunMgr --> DistSampler
    RunMgr --> RandCrop

    IMProv --> DynLayers
    BaseProv --> DynOp
    DLLoader --> ProgShrink
    DLWorker --> ProgShrink
    DistSampler --> ProgShrink
    RandCrop --> ProgShrink

    DynLayers --> ProgShrink
    DynOp --> ProgShrink
    ProgShrink --> ModelZoo
    MBV3 --> ProgShrink
    ProxyNAS --> ProgShrink
    ResNets --> ProgShrink
    ENNUtils --> ProgShrink

    ModelZoo --> AccDS
    ModelZoo --> LatTable
    AccDS --> AccPred
    ArchEnc --> AccPred
    AccPred --> EvolSearch
    LatTable --> EvolSearch
    EvolSearch -->|"generate subnets"| RunMgr

    CommonTools --> ProgShrink
    FlopsCnt --> ProgShrink
    Layers --> DynOp
    PTMods --> ProgShrink
    PTUtils --> ProgShrink

    PyTorch --> ProgShrink
    Horovod --> DistRunMgr
    ImageNetDS --> IMProv
    PythonEnv --> ProgShrink

    %% Click Events
    click Train "https://github.com/mit-han-lab/once-for-all/blob/master/train_ofa_net.py"
    click EvalOFA "https://github.com/mit-han-lab/once-for-all/blob/master/eval_ofa_net.py"
    click EvalSpec "https://github.com/mit-han-lab/once-for-all/blob/master/eval_specialized_net.py"
    click DistRunMgr "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/imagenet_classification/run_manager/distributed_run_manager.py"
    click RunMgr "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/imagenet_classification/run_manager/run_manager.py"
    click RunConfig "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/imagenet_classification/run_manager/run_config.py"
    click IMProv "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/imagenet_classification/data_providers/imagenet.py"
    click BaseProv "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/imagenet_classification/data_providers/base_provider.py"
    click DLLoader "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/utils/my_dataloader/my_data_loader.py"
    click DLWorker "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/utils/my_dataloader/my_data_worker.py"
    click DistSampler "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/utils/my_dataloader/my_distributed_sampler.py"
    click RandCrop "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/utils/my_dataloader/my_random_resize_crop.py"
    click DynLayers "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/imagenet_classification/elastic_nn/modules/dynamic_layers.py"
    click DynOp "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/imagenet_classification/elastic_nn/modules/dynamic_op.py"
    click MBV3 "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/imagenet_classification/elastic_nn/networks/ofa_mbv3.py"
    click ProxyNAS "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/imagenet_classification/elastic_nn/networks/ofa_proxyless.py"
    click ResNets "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/imagenet_classification/elastic_nn/networks/ofa_resnets.py"
    click ProgShrink "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/imagenet_classification/elastic_nn/training/progressive_shrinking.py"
    click ENNUtils "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/imagenet_classification/elastic_nn/utils.py"
    click ModelZoo "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/model_zoo.py"
    click AccDS "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/nas/accuracy_predictor/acc_dataset.py"
    click AccPred "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/nas/accuracy_predictor/acc_predictor.py"
    click ArchEnc "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/nas/accuracy_predictor/arch_encoder.py"
    click LatTable "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/nas/efficiency_predictor/latency_lookup_table.py"
    click EvolSearch "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/nas/search_algorithm/evolution.py"
    click CommonTools "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/utils/common_tools.py"
    click FlopsCnt "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/utils/flops_counter.py"
    click Layers "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/utils/layers.py"
    click PTMods "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/utils/pytorch_modules.py"
    click PTUtils "https://github.com/mit-han-lab/once-for-all/blob/master/ofa/utils/pytorch_utils.py"

    %% Styles
    classDef scripts fill:#bbdefb,stroke:#333,stroke-width:1px
    classDef data fill:#fff59d,stroke:#333,stroke-width:1px
    classDef core fill:#c8e6c9,stroke:#333,stroke-width:1px
    classDef external fill:#ffcc80,stroke:#333,stroke-width:1px
```
