How dandelin/vilt works
Project Type and Purpose
• This is a vision‐and‐language pretraining and inference library (“ViLT”) implementing a Transformer‐only VLP model for tasks like Masked Language Modeling (MLM), Image‐Text Matching (ITM), VQA, IR/TR, etc.
• It uses PyTorch Lightning (via run.py + config.py), Hydra for configuration, and Gradio for interactive demos.
File‐Structure Analysis
• Top‐level scripts:
• run.py – entry point for training/evaluation (Lightning + Hydra).
• demo.py / demo_vqa.py – Gradio‐based inference UIs.
• setup.py, requirements.txt – packaging and dependencies.
• Core package vilt/:
• config.py – Hydra config definitions (model, dataset, training).
• datamodules/ – Lightning DataModules for each dataset (COCO, VQAv2, NLVR2, etc.).
• datasets/ – raw Dataset classes implementing __getitem__ + label loading.
• transforms/ – image/text preprocessing pipelines (pixelbert, randaug).
• modules/ – model definitions:
– vision_transformer.py – pure‐ViT image encoder
– vilt_module.py – LightningModule combining vision & text, heads + losses
– heads.py, objectives.py – task‐specific classification/regression heads
– dist_utils.py, vilt_utils.py – distributed/training helpers
• gadgets/ – custom metric calculations
• utils/ – dataset conversion scripts, glossary.
Key Architectural Patterns
• Modular ML pipeline:
– Configuration (Hydra) → DataModule → Model (LightningModule) → Trainer
• Separation of concerns:
– Data ingestion/preprocessing (datasets + transforms)
– Model building (modules/vision_transformer.py + vilt_module.py)
– Task heads & objectives decoupled for extensibility
• Interactive demo layer (Gradio) separated from training pipeline.
Main Components & Interactions
• Component A: CLI & Config
• run.py loads config.py (Hydra) → instantiates DataModule + LightningModule → PyTorch Lightning Trainer.
• Component B: Data Pipeline
• Raw data files → datasets/*.py → transforms/*.py → DataLoader (in datamodules/*)
• DataModule provides train/val/test_dataloader() to Trainer.
• Component C: Core Model
• Image patches → vision_transformer encoder
• Text tokens → standard token embedding + positional embeddings
• Joint Transformer layers → multimodal fusion
• Task‐specific heads → predictions
• Losses computed in objectives.py
• Component D: Training & Evaluation
• LightningModule (vilt_module.py) implements training_step, validation_step, logging (via gadgets/my_metrics.py), checkpointing.
• Component E: Inference / Demo
• Scripts (demo.py, demo_vqa.py) load pretrained checkpoints → Gradio interface → user inputs image & text queries → model inference → outputs & visualizations.