How compvis/distilldift works
Project Type and Purpose
• DistillDIFT is a research-oriented ML codebase for “Distillation of Diffusion Features for Semantic Correspondence.”
• It comprises two main pipelines:
a) Training pipeline (under the train/ directory) to distill and refine models using various datasets and supervision modes.
b) Evaluation pipeline (under the eval/ directory) to benchmark the distilled models on semantic correspondence benchmarks like SPair-71K.
File-Structure Analysis
• Top-level:
• train/: all training code, configs, data scripts, models, extractors, notebooks.
• eval/: evaluation scripts, configs, checkpoints, model utilities, data prep.
• assets/, README.md, .gitignore.
• Patterns:
• Modular ML pipeline: data preparation → feature extraction → model definition → training loop → logging → checkpointing.
• Two separate entry points: train/train.py (uses accelerate launch) and eval/eval.py.
• Config-driven: YAMLs under train/configs/ and eval/configs/.
• Extensive use of utility modules (train/utils/, eval/utils/) for dataset handling, metrics, visualization, and logging.
Key README Insights
• Purpose: distill two vision foundation models (e.g., DINO, diffusion models) into a unified lightweight semantic correspondence model.
• Supports supervised, weakly supervised, unsupervised modes (distilled_s, distilled_ws, distilled_us).
• Data pipelines for COCO, SPair-71K, CO3D etc., with shell scripts to download and preprocess.
• Evaluation leverages existing geo-aware SC code; results serialized to result.pkl.
Main Components
a. Data Ingestion & Preparation
• Shell scripts (download_*.sh), dataset builders (train/datasets/…, eval/data/prepare_*.sh).
• Output: preprocessed datasets (HDF5, image folders).
b. Feature Extractors
• Pretrained vision models under train/extractors/ and eval/model_utils/ (e.g., DINO, SD, ViT, diffusion).
c. Projection & Distillation Network
• Projection network connects extractor outputs to distilled model (eval/model_utils/projection_network.py, train/models/distilled_model.py).
d. Training Loop
• train/train.py: loads configs, datasets, extractors, distilled model, loss functions (train/utils/distillation.py), optimizer, scheduler.
• Logging & metrics via train/utils/visualization.py and logger.
• Checkpoint saving .ckpt files.
e. Evaluation Pipeline
• eval/eval.py orchestrates loading of distilled checkpoints, calculates correspondence metrics (eval/utils/eval_spair.py), visualization.
• Results stored under eval/results_spair/.
f. Configuration Layer
• YAMLs control hyperparameters, dataset choices, augmentations.
g. External Services / Tools
• PyTorch / Accelerate, Hydra-like config, shell environment for dataset download.
Relationships & Interactions
• Data Prep → Dataset Objects → Dataloader
• Dataloader + Extractors → Feature Maps
• Feature Maps → Projection Network → Distilled Model → Loss Computation → Optimizer
• During inference: Distilled Model + Input Image → Correspondence Map → Evaluation Metrics
• Checkpoints flow from Training to Evaluation.
Architectural Patterns & Principles
• Modular separation: data, extractors, model definitions, training logic, evaluation logic.
• Config-driven design for experiments reproducibility.
• Reusable utility modules for logging, visualization, dataset handling.
• Clear pipeline: Preprocessing → Training → Checkpointing → Evaluation.