# ailab-cvc/m2pt architecture

> Architecture diagram of the GitHub repository ailab-cvc/m2pt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ailab-cvc/m2pt
- Repository: https://github.com/ailab-cvc/m2pt

## Overview

##### Project Type and Purpose
- Research codebase for multimodal transformer training (CVPR 2024).
- Implements a “Multimodal Pathway” that augments a target‐modality ViT with irrelevant data from other modalities.
- Contains four modality modules: Image, Video, Audio, Point-cloud.

##### File Structure
- Top‐level directories: Image/, Video/, Audio/, Point/ – each holds:
– dataset/ (data loaders, transforms)
– models_*.py (MAE, ViT variants)
– engine_pretrain.py & engine_finetune.py (training loops)
– main_*.py (entry-point scripts)
– submitit_*.py or shell scripts (job submission)
– util/ (optimizers, schedulers, misc helpers)
- Root: multimodal_pathway.py (core re-parameterization function), assets/, README.md.
- Configuration & registry: Point/utils/config.py, registry.py, parser.py.

##### Extract Architecture from README
- Core idea: given a target model and an auxiliary model (two ViT backbones), connect corresponding blocks (QKV, projection, MLP) at training time via Cross‐Modal Re-parameterization.
- No extra inference cost; auxiliary weights are merged.
- Supports four tasks (image, video, audio, point clouds) with similar pipelines.

##### Main Components
a. Data Ingestion Layer (per modality)
– Dataset modules: Image/util/datasets.py, Video/dataset/build.py, Audio/dataset.py, Point/datasets/*.py
– Data Transforms & Loader
b. Model Definition Layer
– Target Model: ViT base (models_vit.py or models_mae.py in each folder)
– Aux Model: same architecture, separate checkpoint
c. Cross-Modal Re-parameterization Module
– multimodal_pathway.reparameterize_aux_into_target_model
– Injects auxiliary transformer weights into target model’s transformer blocks
d. Training Engine Layer
– Pretraining Engine: engine_pretrain.py (each modality)
– Finetuning Engine: engine_finetune.py + main_finetune_*.py
– submitit_* or shell scripts (job orchestration)
e. Configuration & Utilities
– Config parser (yaml files under Point/cfgs, or CLI arguments)
– Common Utilities: lr schedulers, optimizers (util/lars.py, util/lr_sched.py)
f. External Dependencies
– PyTorch, timm, submitit

##### Relationships and Data Flow
- Engine instantiates Target Model and Aux Model via timm.create_model, loads checkpoints.
- Engine calls reparameterization to fuse weights.
- Fused Model processes batches through transformer blocks.
- Loss → Backprop → Optimizer → Updated Target Model weights.
- Checkpoint saving → can be used as new pretrained model.

##### Architectural Patterns & Principles
– Modular separation by modality (folder‐based, DRY training logic).
– Strategy pattern for runners (Point/tools/runner*).
– Config‐driven experiments (YAML + CLI).
– Reuse of transformer abstractions across modalities.
– Zero additional inference cost via weight merging (re-parameterization).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        direction TB
        ImgData["Image Dataset"]:::data
        click ImgData "https://github.com/ailab-cvc/m2pt/blob/main/Image/util/datasets.py"
        VidBuild["Video Build Dataset"]:::data
        click VidBuild "https://github.com/ailab-cvc/m2pt/blob/main/Video/dataset/build.py"
        VidData["Video Datasets"]:::data
        click VidData "https://github.com/ailab-cvc/m2pt/blob/main/Video/dataset/datasets.py"
        AudData["Audio Dataset"]:::data
        click AudData "https://github.com/ailab-cvc/m2pt/blob/main/Audio/dataset.py"
        AudUtilData["Audio Util Datasets"]:::data
        click AudUtilData "https://github.com/ailab-cvc/m2pt/blob/main/Audio/util/datasets.py"
        PtBuild["Point-cloud Build"]:::data
        click PtBuild "https://github.com/ailab-cvc/m2pt/blob/main/Point/datasets/build.py"
        PtData["Point-cloud Datasets"]:::data
        click PtData "https://github.com/ailab-cvc/m2pt/blob/main/Point/datasets/ModelNetDataset.py"
    end

    %% Entry Scripts / Orchestration
    subgraph "Entry Scripts & Orchestration"
        direction TB
        ImgMainPre["Image main_pretrain.py"]:::script
        click ImgMainPre "https://github.com/ailab-cvc/m2pt/blob/main/Image/main_pretrain.py"
        ImgMainFine["Image main_finetune.py"]:::script
        click ImgMainFine "https://github.com/ailab-cvc/m2pt/blob/main/Image/main_finetune.py"
        VidPre["Video run_mae_pretraining.py"]:::script
        click VidPre "https://github.com/ailab-cvc/m2pt/blob/main/Video/run_mae_pretraining.py"
        VidFine["Video run_class_finetuning.py"]:::script
        click VidFine "https://github.com/ailab-cvc/m2pt/blob/main/Video/run_class_finetuning.py"
        AudMainPre["Audio main_pretrain.py"]:::script
        click AudMainPre "https://github.com/ailab-cvc/m2pt/blob/main/Audio/main_pretrain.py"
        AudMainFine["Audio main_finetune_as.py"]:::script
        click AudMainFine "https://github.com/ailab-cvc/m2pt/blob/main/Audio/main_finetune_as.py"
        PtMain["Point main.py"]:::script
        click PtMain "https://github.com/ailab-cvc/m2pt/blob/main/Point/main.py"
        PtPathway["Point main_pathway.py"]:::script
        click PtPathway "https://github.com/ailab-cvc/m2pt/blob/main/Point/main_pathway.py"
    end

    %% Model Definition Layer
    subgraph "Model Definition Layer"
        direction TB
        subgraph "Image Models"
            direction TB
            ImgVit["models_vit.py"]:::model
            click ImgVit "https://github.com/ailab-cvc/m2pt/blob/main/Image/models_vit.py"
            ImgMae["models_mae.py"]:::model
            click ImgMae "https://github.com/ailab-cvc/m2pt/blob/main/Image/models_mae.py"
        end
        subgraph "Video Models"
            direction TB
            VidPreModel["modeling_pretrain.py"]:::model
            click VidPreModel "https://github.com/ailab-cvc/m2pt/blob/main/Video/models/modeling_pretrain.py"
            VidFineModel["modeling_finetune.py"]:::model
            click VidFineModel "https://github.com/ailab-cvc/m2pt/blob/main/Video/models/modeling_finetune.py"
        end
        subgraph "Audio Models"
            direction TB
            AudVit["models_vit.py"]:::model
            click AudVit "https://github.com/ailab-cvc/m2pt/blob/main/Audio/models_vit.py"
            AudMae["models_mae.py"]:::model
            click AudMae "https://github.com/ailab-cvc/m2pt/blob/main/Audio/models_mae.py"
        end
        subgraph "Point-cloud Models"
            direction TB
            PtMAE["Point_MAE.py"]:::model
            click PtMAE "https://github.com/ailab-cvc/m2pt/blob/main/Point/models/Point_MAE.py"
            PtBuildModel["build.py"]:::model
            click PtBuildModel "https://github.com/ailab-cvc/m2pt/blob/main/Point/models/build.py"
        end
    end

    %% Cross-Modal Re-parameterization
    Reparam["Cross-Modal Re-parameterization"]:::reparam
    click Reparam "https://github.com/ailab-cvc/m2pt/blob/main/multimodal_pathway.py"

    %% Training Engine Layer
    subgraph "Training Engines"
        direction TB
        ImgEnginePre["engine_pretrain.py"]:::engine
        click ImgEnginePre "https://github.com/ailab-cvc/m2pt/blob/main/Image/engine_pretrain.py"
        ImgEngineFine["engine_finetune.py"]:::engine
        click ImgEngineFine "https://github.com/ailab-cvc/m2pt/blob/main/Image/engine_finetune.py"
        VidEnginePre["engine_for_pretraining.py"]:::engine
        click VidEnginePre "https://github.com/ailab-cvc/m2pt/blob/main/Video/engine_for_pretraining.py"
        VidEngineFine["engine_for_finetuning.py"]:::engine
        click VidEngineFine "https://github.com/ailab-cvc/m2pt/blob/main/Video/engine_for_finetuning.py"
        AudEnginePre["engine_pretrain.py"]:::engine
        click AudEnginePre "https://github.com/ailab-cvc/m2pt/blob/main/Audio/engine_pretrain.py"
        AudEngineFine["engine_finetune.py"]:::engine
        click AudEngineFine "https://github.com/ailab-cvc/m2pt/blob/main/Audio/engine_finetune.py"
        PtRunnerPre["runner_pretrain.py"]:::engine
        click PtRunnerPre "https://github.com/ailab-cvc/m2pt/blob/main/Point/tools/runner_pretrain.py"
        PtRunnerFine["runner_finetune_pathway.py"]:::engine
        click PtRunnerFine "https://github.com/ailab-cvc/m2pt/blob/main/Point/tools/runner_finetune_pathway.py"
    end

    %% Utilities & Configuration
    subgraph "Configuration & Utilities"
        direction TB
        Config["config.yaml / parser / registry"]:::util
        click Config "https://github.com/ailab-cvc/m2pt/blob/main/Point/utils/config.py"
        click Config "https://github.com/ailab-cvc/m2pt/blob/main/Point/utils/parser.py"
        click Config "https://github.com/ailab-cvc/m2pt/blob/main/Point/utils/registry.py"
        click Config "https://github.com/ailab-cvc/m2pt/blob/main/Point/cfgs/*.yaml"
        UtilsCommon["Common Utils (lars, lr_sched, misc)"]:::util
    end

    %% Optimizer & Checkpoint
    Optimizer["Optimizer"]:::util
    Checkpoint["Checkpoint Store"]:::storage
    click Checkpoint "https://github.com/ailab-cvc/m2pt/blob/main/Point/utils/checkpoint.py"

    %% Data Flow Connections
    ImgData --> ImgMainPre --> ImgEnginePre
    ImgEnginePre --> ImgVit --> Reparam
    ImgEnginePre --> ImgMae --> Reparam

    VidBuild --> VidPre --> VidEnginePre
    VidData --> VidPre
    VidEnginePre --> VidPreModel --> Reparam
    VidEnginePre --> VidFineModel --> Reparam

    AudData --> AudMainPre --> AudEnginePre
    AudUtilData --> AudMainPre
    AudEnginePre --> AudVit --> Reparam
    AudEnginePre --> AudMae --> Reparam

    PtBuild --> PtMain --> PtRunnerPre --> PtMAE --> Reparam
    PtData --> PtMain

    %% Reparameterization to Training Loop
    Reparam --> Optimizer
    Optimizer --> Checkpoint

    %% Styles
    classDef data fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef script fill:#ffecb3,stroke:#f9a825,color:#e65100
    classDef model fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20
    classDef engine fill:#ffe0b2,stroke:#fb8c00,color:#e65100
    classDef reparam fill:#e1bee7,stroke:#8e24aa,color:#4a148c
    classDef util fill:#f5f5f5,stroke:#90a4ae,color:#37474f
    classDef storage fill:#b0bec5,stroke:#546e7a,color:#263238
```
