How rezaakb/peft-vit works
Project Type and Purpose
• This is a research-oriented machine-learning pipeline for parameter-efficient fine-tuning of self-supervised Vision Transformers (ViTs) on image classification tasks.
• It uses PyTorch (and PyTorch Lightning via main.py), configurable via YAML files under configs/, to run training (“fit”), evaluation (“test”), linear probing, LoRA, Block Expansion, or full fine-tuning on standard image datasets.
File-Structure Analysis
• Top level
• main.py – entry point with CLI (Lightning Trainer + Hydra-style config)
• configs/ – organized config groups (full, linear, lora, block, add_three) each with dataset- and hyperparameter-specific YAMLs
• src/ – core modules:
– data.py (data loaders, transforms)
– model.py (ViT instantiation, strategy wrappers)
– loss.py, mixup.py, eval_knn.py, utils.py
• requirements.txt, README.md
• This separation suggests a modular CLI-driven pipeline: configuration → data pipeline → model instantiation (with adaptation strategy) → training/evaluation → checkpoints/logs.
README Insights
• Describes four fine-tuning modes: full, linear probing, LoRA, Block Expansion, plus an extra “add_three” strategy.
• Demonstrates usage patterns (python main.py fit --config configs/...) and evaluation commands.
• Shows that models are pretrained ViTs (e.g., vit-b16-224-in21k) anddatasets include CIFAR-10/100, Pets37, Flowers102, Food101, DTD, or custom.
a. Main Components
• CLI & Config Loader (Hydra-like)
• Data Pipeline (datasets, augmentations, DataLoader)
• Model Component (ViT backbone + adaptation strategy plugin)
• Trainer Engine (PyTorch Lightning)
• Loss & Metric Modules (CrossEntropy, mixup, k-NN eval)
• Checkpointing & Logging (TensorBoard or Weights & Biases)
• Storage/Output (config snapshots, model checkpoints, logs)
• External Resources: GPU/CPU, File System, Remote Dataset URLs
b. Relationships & Interactions
1. User invokes main.py CLI → config loader parses selected YAML.
2. Config → instantiates DataModule (src/data.py) → loads dataset, applies transforms.
3. Config + model strategy choice → builds ViT model wrapper (src/model.py) with chosen parameter-efficient method.
4. Model + DataModule → handed to Trainer engine → runs training loops invoking loss (src/loss.py) and optional mixup (src/mixup.py).
5. After each epoch/step → checkpoints & logs written to disk (output/...).
6. Evaluation mode (test) → loads checkpoint + config → runs evaluation via eval_knn.py or Lightning’s test loop.
c. Architectural Patterns
• Modular CLI with Hydra-style config groups
• Data-Module pattern (separation of data loading from model)
• Strategy Pattern in Model (pluggable adapters for full, linear, LoRA, block)
• PyTorch Lightning orchestration (trainer abstraction)
d. Key Technologies
• PyTorch & PyTorch Lightning
• Vision Transformer (Timm or native ViT implementation)
• YAML-based configurations
• Optional external loggers (TensorBoard, WandB)