How facebookresearch/dinov2 works
Purpose and entry points
This is a Python/PyTorch research repository for DINOv2 self-supervised Vision Transformer (ViT) feature learning, pretrained-model consumption, and downstream evaluation. It also adds Cell-DINO, channel-adaptive microscopy models, XRay-DINO, and DINO.txt vision-language inference. Primary entry points are PyTorch Hub (hubconf.py), distributed training (dinov2/run/train/train.py), and evaluation CLIs (dinov2/run/eval/knn.py, dinov2/run/eval/linear.py).
Model foundation
The reusable backbone is a ViT with patch embeddings, attention blocks, optional register tokens, and intermediate feature extraction. Core implementation resides in dinov2/models/vision_transformer.py, composed from dinov2/layers/block.py and dinov2/layers/attention.py. This model boundary produces global and patch-level features consumed by SSL losses, evaluators, and task-specific heads.
Self-supervised training pipeline
dinov2/train/ssl_meta_arch.py defines the student/teacher SSL architecture; dinov2/train/train.py runs optimization and checkpoint-oriented training. Training combines DINO class-token distillation, iBOT masked-patch prediction, and KoLeo regularization from dinov2/loss/dino_clstoken_loss.py, dinov2/loss/ibot_patch_loss.py, and dinov2/loss/koleo_loss.py. The orchestration CLI resolves YAML/overrides, launches distributed workers, and writes teacher checkpoints for downstream evaluation.
Data and augmentation boundary
Dataset specifications are converted into dataset/loader pipelines under dinov2/data/datasets/ and dinov2/data/loaders.py; image transformations, multi-crop augmentation, collation, and masking are isolated in dinov2/data/augmentations.py, dinov2/data/collate.py, and dinov2/data/masking.py. Standard ImageNet and ImageNet-22k coexist with microscopy-specific adapters/datasets such as dinov2/data/datasets/cell_dino/chammi_hpa.py and channel-aware transforms in dinov2/data/cell_dino/transforms.py.
Distributed execution and state
Training targets Linux GPU clusters: PyTorch 2+, CUDA, xFormers, SLURM, and Submitit-style launch flow through dinov2/run/submit.py. Distributed process setup is centralized in dinov2/distributed/__init__.py; large-model sharding support is in dinov2/fsdp/__init__.py. Checkpoint load/save and resume compatibility form the persistence boundary in dinov2/utils/checkpoint.py; cluster/config utilities are in dinov2/utils/cluster.py and dinov2/utils/config.py.
Evaluation and downstream tasks
Feature quality evaluation has k-NN, linear probing, and logistic regression implementations in dinov2/eval/knn.py, dinov2/eval/linear.py, and dinov2/eval/log_regression.py, invoked by matching dinov2/run/eval/ CLIs. Cell-DINO has separate k-NN/linear implementations in dinov2/eval/cell_dino/knn.py and dinov2/eval/cell_dino/linear.py. Dense-task extensions attach ViT features to depth and segmentation stacks: dinov2/eval/depth/models/builder.py, dinov2/eval/segmentation/models/decode_heads/linear_head.py, and Mask2Former integration in dinov2/eval/segmentation_m2f/models/segmentors/encoder_decoder_mask2former.py; these require mmcv/mmsegmentation extras.
Distribution and pretrained interfaces
hubconf.py exposes stable torch.hub.load constructors backed by standard, Cell-DINO, XRay-DINO, depth, classifier, and DINO.txt factories under dinov2/hub/. Key loading paths include dinov2/hub/backbones.py, dinov2/hub/cell_dino/backbones.py, dinov2/hub/xray_dino/backbones.py, and dinov2/hub/text/dinotxt_model.py. Checkpoints are external artifacts fetched from Meta/public URLs or supplied locally; model/license boundaries differ between Apache-licensed base DINOv2 and noncommercial research Cell-DINO/XRay-DINO assets.
Main entry points: vision_transformer.py Vision Transformer · train.py Optimization loop · ssl_meta_arch.py Student-teacher SSL · checkpoint.py Checkpoint persistence
Components 18
ViT foundation
- Vision Transformer · Produces global and patch-level features.
dinov2/models/vision_transformer.py - Attention blocks
dinov2/layers/block.py
SSL training
- Distributed training CLI · Resolves configuration and launches training.
dinov2/run/train/train.py - Cluster launcher · Submitit-style cluster launch flow.
dinov2/run/submit.py - Distributed and FSDP · Process setup and large-model sharding.
dinov2/distributed/__init__.py - Student-teacher SSL · Coordinates student and teacher ViT passes.
dinov2/train/ssl_meta_arch.py - DINO, iBOT, KoLeo losses · Class-token distillation, masked patches, and regularization.
dinov2/loss/dino_clstoken_loss.py - Optimization loop · Optimizes SSL architecture and manages training state.
dinov2/train/train.py - Checkpoint persistence · Saves, resumes, and loads teacher checkpoints.
dinov2/utils/checkpoint.py
Data pipeline
- Datasets and loaders · Builds standard and microscopy dataset loaders.
dinov2/data/loaders.py - Multi-crop augmentation · Applies transforms, collation, and masking.
dinov2/data/augmentations.py - Microscopy adapters · Cell-DINO datasets and channel transforms.
dinov2/data/datasets/cell_dino/chammi_hpa.py
Evaluation and distribution
- PyTorch Hub · Stable torch.hub model interface.
hubconf.py - Model factories · Standard, Cell-DINO, XRay-DINO, and DINO.txt factories.
- External checkpoints · Public or local pretrained weights.
- Evaluation CLIs · k-NN and linear-probe commands.
dinov2/run/eval/knn.py - Feature evaluators · k-NN, linear, logistic, and Cell-DINO evaluation.
dinov2/eval/knn.py - Dense task heads · Depth, linear segmentation, and Mask2Former extensions.
dinov2/eval/depth/models/builder.py
Connections 20
- PyTorch Hub → Model factories: exports
- Model factories → External checkpoints: loads weights
- Model factories → Vision Transformer: constructs
- Attention blocks → Vision Transformer: composes
- Multi-crop augmentation → Datasets and loaders: transforms batches
- Microscopy adapters → Datasets and loaders: extends
- Datasets and loaders → Student-teacher SSL: multi-crop batches
- Distributed training CLI → Cluster launcher: submits
- Distributed training CLI → Optimization loop: starts
- Cluster launcher → Distributed and FSDP: allocates workers
- Distributed and FSDP → Optimization loop: runs
- Student-teacher SSL → Vision Transformer: student and teacher
- Student-teacher SSL → DINO, iBOT, KoLeo losses: computes objectives
- DINO, iBOT, KoLeo losses → Optimization loop: gradients
- Optimization loop → Checkpoint persistence: saves/resumes
- Checkpoint persistence → Feature evaluators: teacher weights
- Evaluation CLIs → Feature evaluators: invokes
- Feature evaluators → Vision Transformer: extracts features
- Dense task heads → Vision Transformer: uses features
- Checkpoint persistence → Model factories: local weights