# compvis/distilldift architecture

> Architecture diagram of the GitHub repository compvis/distilldift, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/compvis/distilldift
- Repository: https://github.com/compvis/distilldift

## Overview

##### Project Type and Purpose
- DistillDIFT is a research-oriented ML codebase for “Distillation of Diffusion Features for Semantic Correspondence.”
- It comprises two main pipelines:
a) Training pipeline (under the `train/` directory) to distill and refine models using various datasets and supervision modes.
b) Evaluation pipeline (under the `eval/` directory) to benchmark the distilled models on semantic correspondence benchmarks like SPair-71K.

##### File-Structure Analysis
- Top-level:
- `train/`: all training code, configs, data scripts, models, extractors, notebooks.
- `eval/`: evaluation scripts, configs, checkpoints, model utilities, data prep.
- `assets/`, `README.md`, `.gitignore`.
- Patterns:
- Modular ML pipeline: data preparation → feature extraction → model definition → training loop → logging → checkpointing.
- Two separate entry points: `train/train.py` (uses `accelerate launch`) and `eval/eval.py`.
- Config-driven: YAMLs under `train/configs/` and `eval/configs/`.
- Extensive use of utility modules (`train/utils/`, `eval/utils/`) for dataset handling, metrics, visualization, and logging.

##### Key README Insights
- Purpose: distill two vision foundation models (e.g., DINO, diffusion models) into a unified lightweight semantic correspondence model.
- Supports supervised, weakly supervised, unsupervised modes (`distilled_s`, `distilled_ws`, `distilled_us`).
- Data pipelines for COCO, SPair-71K, CO3D etc., with shell scripts to download and preprocess.
- Evaluation leverages existing geo-aware SC code; results serialized to `result.pkl`.

##### Main Components
a. Data Ingestion & Preparation
- Shell scripts (`download_*.sh`), dataset builders (`train/datasets/…`, `eval/data/prepare_*.sh`).
- Output: preprocessed datasets (HDF5, image folders).
b. Feature Extractors
- Pretrained vision models under `train/extractors/` and `eval/model_utils/` (e.g., DINO, SD, ViT, diffusion).
c. Projection & Distillation Network
- Projection network connects extractor outputs to distilled model (`eval/model_utils/projection_network.py`, `train/models/distilled_model.py`).
d. Training Loop
- `train/train.py`: loads configs, datasets, extractors, distilled model, loss functions (`train/utils/distillation.py`), optimizer, scheduler.
- Logging & metrics via `train/utils/visualization.py` and `logger`.
- Checkpoint saving `.ckpt` files.
e. Evaluation Pipeline
- `eval/eval.py` orchestrates loading of distilled checkpoints, calculates correspondence metrics (`eval/utils/eval_spair.py`), visualization.
- Results stored under `eval/results_spair/`.
f. Configuration Layer
- YAMLs control hyperparameters, dataset choices, augmentations.
g. External Services / Tools
- PyTorch / Accelerate, Hydra-like config, shell environment for dataset download.

##### Relationships & Interactions
- Data Prep → Dataset Objects → Dataloader
- Dataloader + Extractors → Feature Maps
- Feature Maps → Projection Network → Distilled Model → Loss Computation → Optimizer
- During inference: Distilled Model + Input Image → Correspondence Map → Evaluation Metrics
- Checkpoints flow from Training to Evaluation.

##### Architectural Patterns & Principles
- Modular separation: data, extractors, model definitions, training logic, evaluation logic.
- Config-driven design for experiments reproducibility.
- Reusable utility modules for logging, visualization, dataset handling.
- Clear pipeline: Preprocessing → Training → Checkpointing → Evaluation.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Top Layer: User Scripts
    subgraph "User Scripts"
        direction TB
        TrainScript["bash train.sh"]:::orchestration
        EvalScript["bash eval.sh"]:::orchestration
    end

    %% Configuration Layer
    subgraph "Configuration Layer"
        direction TB
        TrainConfigs["train/configs/"]:::config
        EvalConfigs["eval/configs/"]:::config
    end

    %% Data Preparation
    subgraph "Data Ingestion & Preparation" 
        direction TB
        TrainDataPrep["train/datasets/"]:::data
        EvalDataPrep["eval/data/"]:::data
    end

    %% Feature Extractors
    subgraph "Feature Extractors"
        direction TB
        TrainExtractors["train/extractors/"]:::model
        EvalExtractors["eval/model_utils/"]:::model
    end

    %% Projection & Distillation Network
    subgraph "Distillation Components"
        direction TB
        ProjectionNet["projection_network.py"]:::model
        DistilledModel["distilled_model.py"]:::model
    end

    %% Training
    subgraph "Training Pipeline"
        direction TB
        TrainEntry["train/train.py"]:::orchestration
        TrainUtils["train/utils/"]:::utils
    end

    %% Checkpoint Storage
    Checkpoints["eval/checkpoints/"]:::storage

    %% Evaluation
    subgraph "Evaluation Pipeline"
        direction TB
        EvalEntry["eval/eval.py"]:::orchestration
        EvalUtils["eval/utils/"]:::utils
        Results["eval/results_spair/"]:::storage
    end

    %% Infrastructure
    subgraph "Infrastructure"
        direction TB
        PyTorch["PyTorch/Accelerate"]:::infra
        GPU["GPU Cluster"]:::infra
        FS["File System"]:::infra
        ExtData["External Datasets"]:::infra
    end

    %% Data Flow & Dependencies
    TrainScript -->|launches| TrainEntry
    EvalScript -->|launches| EvalEntry

    TrainConfigs -->|configs| TrainEntry
    EvalConfigs -->|configs| EvalEntry

    ExtData -->|download| TrainDataPrep
    ExtData -->|download| EvalDataPrep
    TrainDataPrep -->|preprocessed data| FS
    EvalDataPrep -->|preprocessed data| FS

    TrainEntry -->|loads data| TrainDataPrep
    TrainEntry -->|uses| TrainExtractors
    TrainEntry -->|imports| DistilledModel
    TrainEntry -->|calls| TrainUtils
    TrainEntry -->|runs on| PyTorch
    TrainEntry -->|runs on| GPU
    TrainEntry -->|saves ckpt| Checkpoints

    TrainExtractors -->|provides features| TrainEntry
    DistilledModel -->|architecture| TrainEntry
    ProjectionNet -->|used in eval| EvalEntry

    EvalEntry -->|loads ckpt| Checkpoints
    EvalEntry -->|loads data| EvalDataPrep
    EvalEntry -->|uses| EvalExtractors
    EvalEntry -->|uses| ProjectionNet
    EvalEntry -->|calls| EvalUtils
    EvalEntry -->|runs on| PyTorch
    EvalEntry -->|runs on| GPU
    EvalEntry -->|writes results| Results

    %% Click Events
    click TrainDataPrep "https://github.com/compvis/distilldift/tree/main/train/datasets/"
    click EvalDataPrep "https://github.com/compvis/distilldift/tree/main/eval/data/"
    click TrainExtractors "https://github.com/compvis/distilldift/tree/main/train/extractors/"
    click EvalExtractors "https://github.com/compvis/distilldift/tree/main/eval/model_utils/"
    click ProjectionNet "https://github.com/compvis/distilldift/blob/main/eval/model_utils/projection_network.py"
    click DistilledModel "https://github.com/compvis/distilldift/blob/main/train/models/distilled_model.py"
    click TrainEntry "https://github.com/compvis/distilldift/blob/main/train/train.py"
    click TrainUtils "https://github.com/compvis/distilldift/tree/main/train/utils/"
    click Checkpoints "https://github.com/compvis/distilldift/tree/main/eval/checkpoints/"
    click EvalEntry "https://github.com/compvis/distilldift/blob/main/eval/eval.py"
    click EvalUtils "https://github.com/compvis/distilldift/tree/main/eval/utils/"
    click Results "https://github.com/compvis/distilldift/tree/main/eval/results_spair/"
    click TrainConfigs "https://github.com/compvis/distilldift/tree/main/train/configs/"
    click EvalConfigs "https://github.com/compvis/distilldift/tree/main/eval/configs/"

    %% Styles
    classDef data fill:#AED6F1,stroke:#1B4F72,color:#1B2631;
    classDef model fill:#ABEBC6,stroke:#196F3D,color:#145A32;
    classDef orchestration fill:#F9E79F,stroke:#B7950B,color:#7D6608;
    classDef utils fill:#D7BDE2,stroke:#6C3483,color:#4A235A;
    classDef config fill:#F5B7B1,stroke:#C0392B,color:#78281F;
    classDef storage fill:#A9DFBF,stroke:#145A32,color:#0B5345;
    classDef infra fill:#D5D8DC,stroke:#566573,color:#212F3D;
```
