# apple/ml-fastvit architecture

> Architecture diagram of the GitHub repository apple/ml-fastvit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apple/ml-fastvit
- Repository: https://github.com/apple/ml-fastvit

## Overview

##### Project Type and Purpose
- This is an open‐source research codebase for defining, training, evaluating, and exporting FastViT vision transformer models on ImageNet.
- It provides three main CLI entrypoints—train.py, validate.py, and export_model.py—and a model zoo under models/ for hybrid ViT architectures with structural reparameterization.

##### High‐Level Components
a. CLI Entrypoints
- train.py – orchestrates distributed data loading, model instantiation, training loop, checkpointing, and logging.
- validate.py – runs evaluation on a dataset split, loads a checkpoint (fused or unfused), computes metrics.
- export_model.py – converts a fused PyTorch checkpoint into a CoreML package.
b. Model Library (models/)
- fastvit.py – high‐level model definitions.
- modules/ – building blocks (MobileOne, ReplkNet, etc.) and reparameterization logic.
c. Utility Helpers (misc/)
- cosine_annealing.py – learning‐rate scheduler.
- distillation_loss.py – teacher‐student loss for knowledge distillation.
d. External Dependencies
- PyTorch, torchvision, timm – underlying ML framework.
- Dataset (ImageNet) – external file tree.
- CoreML exporter – for model export.
e. Output Artifacts
- Checkpoints (.pth.tar), fused checkpoints for inference.
- Logs/metrics (console or external logger).
- CoreML packages (.mlpackage.zip).

##### Relationships and Data Flow
- User invokes one of the CLI scripts.
- Script initializes configuration (args, hyperparameters).
- DataLoader loads ImageNet images → batches → GPU.
- Model factory (timm.create_model + models.fastvit) builds the network.
- Training loop uses misc utilities (LR scheduling, distillation) → forward/backward → optimizer → writes checkpoint.
- Validation reuses DataLoader + model + checkpoint → computes accuracy/latency.
- Export runner loads fused model → CoreML converter → outputs .mlpackage.

##### Architectural Patterns & Design Principles
- Modular separation: CLI logic, model definitions, loss/scheduler utilities.
- Plug‐and‐play model zoo: adding new variants only requires placing new modules under models/.
- Standard PyTorch training pipeline (Dataset → DataLoader → Model → Optimizer → Scheduler).
- Structural reparameterization: build‐time vs. run‐time graph transformations in modules/.

##### Detailed Capture of Essential Elements
- That train.py imports models.fastvit and misc schedulers.
- That timm.create_model is used to fetch a variant by name.
- Call out pre‐training vs. fine‐tuning paths (unfused vs. reparameterized).

- Top layer: CLI and config
- Middle layer: Data & Model pipelines
- Bottom layer: Utilities, external frameworks, and outputs

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top Layer: CLI & Config
    subgraph "CLI & Configuration"
        direction TB
        TrainCLI["train.py"]:::entrypoint
        ValidateCLI["validate.py"]:::entrypoint
        ExportCLI["export_model.py"]:::entrypoint
        Config["Config (args & hyperparams)"]:::core
    end

    %% Middle Layer: Data & Model Pipeline
    subgraph "Data & Model Pipeline"
        direction TB
        DistLauncher["Distributed Launcher"]:::core
        DataLoader["DataLoader (ImageNet)"]:::core
        ModelFactory["Model Factory (timm.create_model + FastViT)"]:::core
        TrainingLoop["Training Loop"]:::core
        ValidateLoop["Validation Loop"]:::core
        Exporter["CoreML Exporter"]:::core
    end

    %% Bottom Layer: Utilities, External & Outputs
    subgraph "Utilities, External & Outputs"
        direction TB
        UtilsPkg["misc/"]:::utilities
        CosineScheduler["cosine_annealing.py"]:::utilities
        DistillationLoss["distillation_loss.py"]:::utilities
        ModelPkgInit["models/__init__.py"]:::modeldef
        ModulesPkgInit["models/modules/__init__.py"]:::modeldef
        FastViTDef["fastvit.py"]:::modeldef
        MobileOneModule["MobileOne Block"]:::modeldef
        RepLKNetModule["RepLKNet Block"]:::modeldef
        TimmLib["timm Library"]:::external
        TorchLib["PyTorch & torchvision"]:::external
        ExternalDataset["External Dataset (ImageNet FS)"]:::external
        CheckpointWriter["Checkpoint Writer (.pth.tar)"]:::output
        Logs["Logs & Metrics"]:::output
        CoreMLPkg["CoreML Package"]:::output
        ReqTxt["requirements.txt"]:::external
        DocsDir["docs/"]:::external
    end

    %% Connections: Training Flow
    TrainCLI -->|init| Config
    Config --> DistLauncher
    DistLauncher --> DataLoader
    DataLoader --> ModelFactory
    ModelFactory --> TrainingLoop
    TrainingLoop --> CheckpointWriter
    TrainingLoop --> Logs

    %% Utility usage in Training
    TrainingLoop -->|uses| CosineScheduler
    TrainingLoop -->|uses| DistillationLoss
    ModelFactory --> FastViTDef
    FastViTDef --> ModulesPkgInit
    ModulesPkgInit --> MobileOneModule
    ModulesPkgInit --> RepLKNetModule
    ModelPkgInit --> FastViTDef

    %% Connections: Validation Flow
    ValidateCLI -->|init| Config
    Config --> DataLoader
    DataLoader --> ModelFactory
    ModelFactory --> ValidateLoop
    ValidateLoop --> Logs

    %% Connections: Export Flow
    ExportCLI -->|init| Config
    Config --> ModelFactory
    ModelFactory --> Exporter
    Exporter --> CoreMLPkg

    %% External dependencies
    DataLoader --> ExternalDataset
    ModelFactory --> TimmLib
    ModelFactory --> TorchLib
    TrainCLI --> ReqTxt
    ValidateCLI --> ReqTxt
    ExportCLI --> ReqTxt
    TrainCLI --> DocsDir

    %% Click Events
    click TrainCLI "https://github.com/apple/ml-fastvit/blob/main/train.py"
    click ValidateCLI "https://github.com/apple/ml-fastvit/blob/main/validate.py"
    click ExportCLI "https://github.com/apple/ml-fastvit/blob/main/export_model.py"
    click FastViTDef "https://github.com/apple/ml-fastvit/blob/main/models/fastvit.py"
    click MobileOneModule "https://github.com/apple/ml-fastvit/blob/main/models/modules/mobileone.py"
    click RepLKNetModule "https://github.com/apple/ml-fastvit/blob/main/models/modules/replknet.py"
    click ModulesPkgInit "https://github.com/apple/ml-fastvit/blob/main/models/modules/__init__.py"
    click ModelPkgInit "https://github.com/apple/ml-fastvit/blob/main/models/__init__.py"
    click CosineScheduler "https://github.com/apple/ml-fastvit/blob/main/misc/cosine_annealing.py"
    click DistillationLoss "https://github.com/apple/ml-fastvit/blob/main/misc/distillation_loss.py"
    click UtilsPkg "https://github.com/apple/ml-fastvit/blob/main/misc/__init__.py"
    click ReqTxt "https://github.com/apple/ml-fastvit/blob/main/requirements.txt"
    click DocsDir "https://github.com/apple/ml-fastvit/tree/main/docs/"

    %% Styles
    classDef entrypoint fill:#cce5ff,stroke:#3399ff,color:#000;
    classDef core fill:#d4edda,stroke:#28a745,color:#000;
    classDef modeldef fill:#ffe5b4,stroke:#ff8c00,color:#000;
    classDef utilities fill:#f2f2f2,stroke:#999999,color:#000;
    classDef external fill:#e5ccff,stroke:#8000ff,color:#000;
    classDef output fill:#fff3cd,stroke:#ffcd00,color:#000;
```
