# opengvlab/videomaev2 architecture

> Architecture diagram of the GitHub repository opengvlab/videomaev2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opengvlab/videomaev2
- Repository: https://github.com/opengvlab/videomaev2

## Overview

##### Project Type and Purpose
- This is an open-source research codebase implementing “VideoMAE V2,” a video masked autoencoder for self-supervised pre-training and downstream fine-tuning on video action recognition and localization tasks.
- It provides data preparation, dual masking strategies, model definitions (encoder/decoder and classifier heads), training engines, feature extraction utilities, and shell scripts to run experiments at scale.

##### Top-Level Components
- dataset/          ← data ingestion & preprocessing
- models/           ← model architectures (pretrain & finetune)
- engine_for_pretraining.py
- engine_for_finetuning.py
- extract_tad_feature.py
- run_mae_pretraining.py
- run_class_finetuning.py
- scripts/          ← experiment wrappers (shell scripts for clusters)
- misc/             ← configuration maps and label files
- docs/             ← user manuals, dataset and training instructions
- optim_factory.py  ← optimizer & scheduler factory
- utils.py          ← shared utilities (logging, checkpointing, argument parsing)

##### Core Architectural Patterns
- Modular training engines (pretrain vs. finetune) driven by main entry points (run_*.py).
- Data-Pipeline abstraction via dataset/__init__.py and transforms, offering pluggable masking, augmentation, and loader strategies.
- Separation of concerns:
- Data preprocessing (dataset/)
- Model definition (models/)
- Training loop & optimization (engine_*.py + optim_factory.py)
- Experiment orchestration (scripts/ + run_*.py)

1. Data Storage & Preparation (blue)
- Raw video datasets on disk
- dataset/build.py & loader.py
- masking_generator.py (dual masking logic)
- transforms.py, video_transforms.py, rand_augment, random_erasing
2. Data Loader (blue)
- PyTorch DataLoader configured in dataset/__init__.py
3. Training Engines (orange)
- engine_for_pretraining.py
- engine_for_finetuning.py
4. Model Zoo (green)
- models/modeling_pretrain.py (ViT encoder + decoder)
- models/modeling_finetune.py (encoder + classification head)
5. Optimization & Logging (purple)
- optim_factory.py (LR schedulers, optimizers)
- utils.py (checkpoint, tensorboard, console logs)
6. Experiment Orchestration (grey)
- run_mae_pretraining.py
- run_class_finetuning.py
- scripts/pretrain/*.sh
- scripts/finetune/*.sh
7. Feature Extraction (teal)
- extract_tad_feature.py (for temporal action detection features)

1. Data Storage → Data Preparation → Data Loader → Training Engine
2. Training Engine ↔ Model Zoo (forward & backprop calls)
3. Training Engine ↔ Optimization & Logging (loss computation → optimizer updates → checkpoint & metrics logging)
4. Experiment Orchestration → Training Engine (CLI params, config files)
5. Feature Extraction ←– Data Loader + Pretrained Model → extract_tad_feature.py

##### Architectural Highlights & Patterns
- Dual-masking module sits early in the data pipeline, generating spatio-temporal masks for VideoMAE pretraining.
- Two distinct model flows:
- Pretrain: Autoencoder structure with encoder/decoder in modeling_pretrain.py
- Finetune: Encoder with classification head in modeling_finetune.py
- Optimizer factory abstracts optimizer & scheduler instantiation based on configuration.
- Shell scripts in scripts/ provide reproducible launch commands for different model scales and datasets.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Storage & Preparation (blue)
    subgraph "Data Storage & Preparation"
        direction TB
        DS_Raw["Raw Video Datasets (Disk)"]:::data
        Build["build.py"]:::data
        LoaderFile["loader.py"]:::data
        MaskGen["masking_generator.py"]:::data
        Transforms["transforms.py"]:::data
        VideoTransforms["video_transforms.py"]:::data
        RandAug["rand_augment.py"]:::data
        RandErase["random_erasing.py"]:::data
        Functional["functional.py"]:::data
        PretrainDatasets["pretrain_datasets.py"]:::data
        Datasets["datasets.py"]:::data
        VolumeTransforms["volume_transforms.py"]:::data
    end

    %% Data Loader Abstraction
    DL["dataset/__init__.py"]:::data

    %% Training Engines (orange)
    subgraph "Training Engines"
        direction TB
        EnginePre["engine_for_pretraining.py"]:::engine
        EngineFine["engine_for_finetuning.py"]:::engine
    end

    %% Model Zoo (green)
    subgraph "Model Zoo"
        direction TB
        ModelPre["modeling_pretrain.py"]:::model
        ModelFine["modeling_finetune.py"]:::model
    end

    %% Optimization & Logging (purple)
    subgraph "Optimization & Logging"
        direction TB
        OptFactory["optim_factory.py"]:::optlog
        Utils["utils.py"]:::optlog
    end

    %% Experiment Orchestration (grey)
    subgraph "Experiment Orchestration"
        direction TB
        RunPre["run_mae_pretraining.py"]:::orch
        RunFine["run_class_finetuning.py"]:::orch
        ScriptsPre["scripts/pretrain/*.sh"]:::orch
        ScriptsFine["scripts/finetune/*.sh"]:::orch
    end

    %% Feature Extraction (teal)
    FE["extract_tad_feature.py"]:::feat

    %% External Dependencies
    subgraph "External"
        direction TB
        PyTorch["PyTorch / torchvision / mmcv"]:::external
    end

    %% Connections
    DS_Raw --> Build
    Build --> LoaderFile
    LoaderFile --> MaskGen
    MaskGen --> Transforms
    Transforms --> VideoTransforms
    VideoTransforms --> RandAug
    RandAug --> RandErase
    RandErase --> Functional
    Functional --> PretrainDatasets
    PretrainDatasets --> Datasets
    Datasets --> VolumeTransforms
    VolumeTransforms --> DL

    DL --> EnginePre
    DL --> EngineFine

    EnginePre --> ModelPre
    EngineFine --> ModelFine
    ModelPre --> EnginePre
    ModelFine --> EngineFine

    EnginePre --> OptFactory
    EngineFine --> OptFactory
    OptFactory --> EnginePre
    OptFactory --> EngineFine
    EnginePre --> Utils
    EngineFine --> Utils

    RunPre --> EnginePre
    RunFine --> EngineFine
    ScriptsPre --> RunPre
    ScriptsFine --> RunFine

    DL --> FE
    ModelPre --> FE
    FE -->|"extract features"| Utils

    PyTorch --> DL
    PyTorch --> ModelPre
    PyTorch --> ModelFine
    PyTorch --> EnginePre
    PyTorch --> EngineFine

    %% Click Events
    click Build "https://github.com/opengvlab/videomaev2/blob/master/dataset/build.py"
    click LoaderFile "https://github.com/opengvlab/videomaev2/blob/master/dataset/loader.py"
    click MaskGen "https://github.com/opengvlab/videomaev2/blob/master/dataset/masking_generator.py"
    click Transforms "https://github.com/opengvlab/videomaev2/blob/master/dataset/transforms.py"
    click VideoTransforms "https://github.com/opengvlab/videomaev2/blob/master/dataset/video_transforms.py"
    click RandAug "https://github.com/opengvlab/videomaev2/blob/master/dataset/rand_augment.py"
    click RandErase "https://github.com/opengvlab/videomaev2/blob/master/dataset/random_erasing.py"
    click Functional "https://github.com/opengvlab/videomaev2/blob/master/dataset/functional.py"
    click PretrainDatasets "https://github.com/opengvlab/videomaev2/blob/master/dataset/pretrain_datasets.py"
    click Datasets "https://github.com/opengvlab/videomaev2/blob/master/dataset/datasets.py"
    click VolumeTransforms "https://github.com/opengvlab/videomaev2/blob/master/dataset/volume_transforms.py"
    click DL "https://github.com/opengvlab/videomaev2/blob/master/dataset/__init__.py"
    click EnginePre "https://github.com/opengvlab/videomaev2/blob/master/engine_for_pretraining.py"
    click EngineFine "https://github.com/opengvlab/videomaev2/blob/master/engine_for_finetuning.py"
    click ModelPre "https://github.com/opengvlab/videomaev2/blob/master/models/modeling_pretrain.py"
    click ModelFine "https://github.com/opengvlab/videomaev2/blob/master/models/modeling_finetune.py"
    click OptFactory "https://github.com/opengvlab/videomaev2/blob/master/optim_factory.py"
    click Utils "https://github.com/opengvlab/videomaev2/blob/master/utils.py"
    click RunPre "https://github.com/opengvlab/videomaev2/blob/master/run_mae_pretraining.py"
    click RunFine "https://github.com/opengvlab/videomaev2/blob/master/run_class_finetuning.py"
    click ScriptsPre "https://github.com/opengvlab/videomaev2/tree/master/scripts/pretrain/"
    click ScriptsFine "https://github.com/opengvlab/videomaev2/tree/master/scripts/finetune/"
    click FE "https://github.com/opengvlab/videomaev2/blob/master/extract_tad_feature.py"

    %% Styles
    classDef data fill:#cce5ff,stroke:#005cbf,color:#005cbf;
    classDef engine fill:#ffe5cc,stroke:#ff8000,color:#ff8000;
    classDef model fill:#d4edda,stroke:#28a745,color:#28a745;
    classDef optlog fill:#e2ccff,stroke:#8000ff,color:#8000ff;
    classDef orch fill:#f0f0f0,stroke:#6c757d,color:#6c757d;
    classDef feat fill:#d1ecf1,stroke:#17a2b8,color:#17a2b8;
    classDef external fill:#ffffff,stroke:#343a40,stroke-dasharray: 5 5,color:#343a40;
```
