How ailab-cvc/m2pt works
Project Type and Purpose
• Research codebase for multimodal transformer training (CVPR 2024).
• Implements a “Multimodal Pathway” that augments a target‐modality ViT with irrelevant data from other modalities.
• Contains four modality modules: Image, Video, Audio, Point-cloud.
File Structure
• Top‐level directories: Image/, Video/, Audio/, Point/ – each holds:
– dataset/ (data loaders, transforms)
– models_*.py (MAE, ViT variants)
– engine_pretrain.py & engine_finetune.py (training loops)
– main_*.py (entry-point scripts)
– submitit_*.py or shell scripts (job submission)
– util/ (optimizers, schedulers, misc helpers)
• Root: multimodal_pathway.py (core re-parameterization function), assets/, README.md.
• Configuration & registry: Point/utils/config.py, registry.py, parser.py.
Extract Architecture from README
• Core idea: given a target model and an auxiliary model (two ViT backbones), connect corresponding blocks (QKV, projection, MLP) at training time via Cross‐Modal Re-parameterization.
• No extra inference cost; auxiliary weights are merged.
• Supports four tasks (image, video, audio, point clouds) with similar pipelines.
Main Components
a. Data Ingestion Layer (per modality)
– Dataset modules: Image/util/datasets.py, Video/dataset/build.py, Audio/dataset.py, Point/datasets/*.py
– Data Transforms & Loader
b. Model Definition Layer
– Target Model: ViT base (models_vit.py or models_mae.py in each folder)
– Aux Model: same architecture, separate checkpoint
c. Cross-Modal Re-parameterization Module
– multimodal_pathway.reparameterize_aux_into_target_model
– Injects auxiliary transformer weights into target model’s transformer blocks
d. Training Engine Layer
– Pretraining Engine: engine_pretrain.py (each modality)
– Finetuning Engine: engine_finetune.py + main_finetune_*.py
– submitit_* or shell scripts (job orchestration)
e. Configuration & Utilities
– Config parser (yaml files under Point/cfgs, or CLI arguments)
– Common Utilities: lr schedulers, optimizers (util/lars.py, util/lr_sched.py)
f. External Dependencies
– PyTorch, timm, submitit
Relationships and Data Flow
• Engine instantiates Target Model and Aux Model via timm.create_model, loads checkpoints.
• Engine calls reparameterization to fuse weights.
• Fused Model processes batches through transformer blocks.
• Loss → Backprop → Optimizer → Updated Target Model weights.
• Checkpoint saving → can be used as new pretrained model.
Architectural Patterns & Principles
– Modular separation by modality (folder‐based, DRY training logic).
– Strategy pattern for runners (Point/tools/runner*).
– Config‐driven experiments (YAML + CLI).
– Reuse of transformer abstractions across modalities.
– Zero additional inference cost via weight merging (re-parameterization).