How apple/ml-fastvit works
Project Type and Purpose
• This is an open‐source research codebase for defining, training, evaluating, and exporting FastViT vision transformer models on ImageNet.
• It provides three main CLI entrypoints—train.py, validate.py, and export_model.py—and a model zoo under models/ for hybrid ViT architectures with structural reparameterization.
High‐Level Components
a. CLI Entrypoints
• train.py – orchestrates distributed data loading, model instantiation, training loop, checkpointing, and logging.
• validate.py – runs evaluation on a dataset split, loads a checkpoint (fused or unfused), computes metrics.
• export_model.py – converts a fused PyTorch checkpoint into a CoreML package.
b. Model Library (models/)
• fastvit.py – high‐level model definitions.
• modules/ – building blocks (MobileOne, ReplkNet, etc.) and reparameterization logic.
c. Utility Helpers (misc/)
• cosine_annealing.py – learning‐rate scheduler.
• distillation_loss.py – teacher‐student loss for knowledge distillation.
d. External Dependencies
• PyTorch, torchvision, timm – underlying ML framework.
• Dataset (ImageNet) – external file tree.
• CoreML exporter – for model export.
e. Output Artifacts
• Checkpoints (.pth.tar), fused checkpoints for inference.
• Logs/metrics (console or external logger).
• CoreML packages (.mlpackage.zip).
Relationships and Data Flow
• User invokes one of the CLI scripts.
• Script initializes configuration (args, hyperparameters).
• DataLoader loads ImageNet images → batches → GPU.
• Model factory (timm.create_model + models.fastvit) builds the network.
• Training loop uses misc utilities (LR scheduling, distillation) → forward/backward → optimizer → writes checkpoint.
• Validation reuses DataLoader + model + checkpoint → computes accuracy/latency.
• Export runner loads fused model → CoreML converter → outputs .mlpackage.
Architectural Patterns & Design Principles
• Modular separation: CLI logic, model definitions, loss/scheduler utilities.
• Plug‐and‐play model zoo: adding new variants only requires placing new modules under models/.
• Standard PyTorch training pipeline (Dataset → DataLoader → Model → Optimizer → Scheduler).
• Structural reparameterization: build‐time vs. run‐time graph transformations in modules/.
Detailed Capture of Essential Elements
• That train.py imports models.fastvit and misc schedulers.
• That timm.create_model is used to fetch a variant by name.
• Call out pre‐training vs. fine‐tuning paths (unfused vs. reparameterized).
• Top layer: CLI and config
• Middle layer: Data & Model pipelines
• Bottom layer: Utilities, external frameworks, and outputs