# dandelin/vilt architecture

> Architecture diagram of the GitHub repository dandelin/vilt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dandelin/vilt
- Repository: https://github.com/dandelin/vilt

## Overview

##### Project Type and Purpose
- This is a vision‐and‐language pretraining and inference library (“ViLT”) implementing a Transformer‐only VLP model for tasks like Masked Language Modeling (MLM), Image‐Text Matching (ITM), VQA, IR/TR, etc.
- It uses PyTorch Lightning (via `run.py` + `config.py`), Hydra for configuration, and Gradio for interactive demos.

##### File‐Structure Analysis
- Top‐level scripts:
- `run.py` – entry point for training/evaluation (Lightning + Hydra).
- `demo.py` / `demo_vqa.py` – Gradio‐based inference UIs.
- `setup.py`, `requirements.txt` – packaging and dependencies.
- Core package `vilt/`:
- `config.py` – Hydra config definitions (model, dataset, training).
- `datamodules/` – Lightning DataModules for each dataset (COCO, VQAv2, NLVR2, etc.).
- `datasets/` – raw Dataset classes implementing `__getitem__` + label loading.
- `transforms/` – image/text preprocessing pipelines (pixelbert, randaug).
- `modules/` – model definitions:
– `vision_transformer.py` – pure‐ViT image encoder
– `vilt_module.py` – LightningModule combining vision & text, heads + losses
– `heads.py`, `objectives.py` – task‐specific classification/regression heads
– `dist_utils.py`, `vilt_utils.py` – distributed/training helpers
- `gadgets/` – custom metric calculations
- `utils/` – dataset conversion scripts, glossary.

##### Key Architectural Patterns
- Modular ML pipeline:
– Configuration (Hydra) → DataModule → Model (LightningModule) → Trainer
- Separation of concerns:
– Data ingestion/preprocessing (`datasets` + `transforms`)
– Model building (`modules/vision_transformer.py` + `vilt_module.py`)
– Task heads & objectives decoupled for extensibility
- Interactive demo layer (Gradio) separated from training pipeline.

##### Main Components & Interactions
- Component A: CLI & Config
- `run.py` loads `config.py` (Hydra) → instantiates DataModule + LightningModule → PyTorch Lightning `Trainer`.
- Component B: Data Pipeline
- Raw data files → `datasets/*.py` → `transforms/*.py` → DataLoader (in `datamodules/*`)
- DataModule provides `train/val/test_dataloader()` to Trainer.
- Component C: Core Model
- Image patches → `vision_transformer` encoder
- Text tokens → standard token embedding + positional embeddings
- Joint Transformer layers → multimodal fusion
- Task‐specific heads → predictions
- Losses computed in `objectives.py`
- Component D: Training & Evaluation
- LightningModule (`vilt_module.py`) implements `training_step`, `validation_step`, logging (via `gadgets/my_metrics.py`), checkpointing.
- Component E: Inference / Demo
- Scripts (`demo.py`, `demo_vqa.py`) load pretrained checkpoints → Gradio interface → user inputs image & text queries → model inference → outputs & visualizations.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "CLI & Config Layer"
        CLI["run.py"]:::cli
        Config["config.py"]:::cli
    end
    subgraph "Data Layer"
        Datasets["Dataset Classes"]:::data
        Transforms["Transforms"]:::data
    end
    subgraph "DataModule Layer"
        DataModule["DataModules"]:::data
    end
    subgraph "Core Model Layer"
        Vision["Vision Transformer"]:::model
        Multimodal["Multimodal LightningModule"]:::model
    end
    Heads["Task Heads"]:::model
    Losses["Loss / Objectives"]:::model
    subgraph "Training Helpers & Utilities"
        Dist["Distributed Utils"]:::training
        Utils["General Utils"]:::training
    end
    Metrics["Metrics & Logging"]:::training
    Trainer["PyTorch Lightning Trainer"]:::training
    DatasetsStore[(Raw Datasets)]:::dataStorage
    Checkpoints[(Checkpoints)]:::dataStorage
    subgraph "Inference UI"
        Demo["demo.py"]:::demo
        DemoVQA["demo_vqa.py"]:::demo
    end

    CLI -->|"loads"| Config
    Config -->|"instantiates"| DataModule
    Config -->|"instantiates"| Multimodal
    DatasetsStore -->|"raw data"| Datasets
    Datasets -->|"__getitem__"| DataModule
    Transforms -->|"applies preprocessing"| DataModule
    DataModule -->|"DataLoader produces batches"| Multimodal
    Vision -->|"encodes images"| Multimodal
    Multimodal -->|"fusion features"| Heads
    Heads -->|"predictions"| Losses
    Losses -->|"computed loss"| Trainer
    Dist --> Trainer
    Utils --> Trainer
    Trainer -->|"saves"| Checkpoints
    Trainer -->|"logs metrics"| Metrics

    Demo -.->|"loads checkpoint"| Checkpoints
    DemoVQA -.->|"loads checkpoint"| Checkpoints
    Demo -.->|"inference"| Multimodal
    DemoVQA -.->|"inference"| Multimodal
    Multimodal -.->|"predictions"| Demo
    Multimodal -.->|"predictions"| DemoVQA

    click CLI "https://github.com/dandelin/vilt/blob/master/run.py"
    click Config "https://github.com/dandelin/vilt/blob/master/vilt/config.py"
    click Datasets "https://github.com/dandelin/vilt/blob/master/vilt/datasets/base_dataset.py"
    click Transforms "https://github.com/dandelin/vilt/blob/master/vilt/transforms/pixelbert.py"
    click DataModule "https://github.com/dandelin/vilt/blob/master/vilt/datamodules/datamodule_base.py"
    click Vision "https://github.com/dandelin/vilt/blob/master/vilt/modules/vision_transformer.py"
    click Multimodal "https://github.com/dandelin/vilt/blob/master/vilt/modules/vilt_module.py"
    click Heads "https://github.com/dandelin/vilt/blob/master/vilt/modules/heads.py"
    click Losses "https://github.com/dandelin/vilt/blob/master/vilt/modules/objectives.py"
    click Dist "https://github.com/dandelin/vilt/blob/master/vilt/modules/dist_utils.py"
    click Utils "https://github.com/dandelin/vilt/blob/master/vilt/modules/vilt_utils.py"
    click Metrics "https://github.com/dandelin/vilt/blob/master/vilt/gadgets/my_metrics.py"
    click Demo "https://github.com/dandelin/vilt/blob/master/demo.py"
    click DemoVQA "https://github.com/dandelin/vilt/blob/master/demo_vqa.py"

    classDef cli fill:#f9f,stroke:#333,stroke-width:1px;
    classDef data fill:#bbf,stroke:#333,stroke-width:1px;
    classDef model fill:#bfb,stroke:#333,stroke-width:1px;
    classDef training fill:#fbf,stroke:#333,stroke-width:1px;
    classDef demo fill:#fbb,stroke:#333,stroke-width:1px;
    classDef dataStorage fill:#ffb,stroke:#333,stroke-width:1px;
```
