# mcg-nju/videomae architecture

> Architecture diagram of the GitHub repository mcg-nju/videomae, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mcg-nju/videomae
- Repository: https://github.com/mcg-nju/videomae

## Overview

##### Project Type and Purpose
- This is an open-source PyTorch research library for self-supervised video pre-training (VideoMAE) and downstream fine-tuning.
- It implements a masked autoencoder on video sequences (plain ViT backbone) with extremely high masking ratios, supporting both pre-training and fine-tuning pipelines.

##### Top-Level Components

2.1 Data Ingestion & Preparation (Blue rectangle)
- DATASET.md, datasets.py, kinetics.py, ssv2.py
- Responsibilities: loading video datasets (Kinetics-400, Something-Something V2, UCF101), parsing metadata, producing raw video frames.

2.2 Data Augmentation & Masking (Blue rectangle)
- transforms.py, video_transforms.py, volume_transforms.py, rand_augment.py, random_erasing.py, mixup.py, masking_generator.py
- Responsibilities: spatial/temporal transforms, random erasing/mixup, tube masking generator for patch masking.

2.3 Model Definitions (Green rectangle)
- modeling_pretrain.py, modeling_finetune.py, functional.py
- Responsibilities:
- Encoder (ViT patch embedding, transformer blocks)
- Decoder (lightweight transformer for reconstruction in pre-training)
- Head for classification in fine-tuning

2.4 Training & Evaluation Engines (Orange rectangle)
- engine_for_pretraining.py, engine_for_finetuning.py
- Responsibilities: training loops, loss computation, backprop, metrics logging, checkpoint I/O.

2.5 Optimization & Config Factory (Yellow rectangle)
- optim_factory.py
- Responsibilities: optimizer and scheduler set-up (AdamW, weight decay, learning-rate schedules).

2.6 Orchestration Scripts (Gray rectangle)
- run_mae_pretraining.py, run_class_finetuning.py, run_videomae_vis.py, vis.sh
- Under scripts/… : dataset-specific shell scripts (pretrain.sh, finetune.sh) for Kinetics, SSV2, UCF101.
- Responsibilities: CLI interfaces, command-line argument parsing, setting up experiment folders, invoking engines with correct configs.

2.7 Utilities & Logging (Light-gray rectangle)
- utils.py
- Responsibilities: helper functions (checkpoint saving/loading, metric tracking, distributed training utilities).

- README.md, PRETRAIN.md, FINETUNE.md, INSTALL.md, DATASET.md, MODEL_ZOO.md
- Responsibilities: user instructions, dataset preparation, installation, links to pre-trained weights.

2.9 External Services (Purple cloud)
- Hugging Face Models/Spaces, PapersWithCode, external dataset links, license references.

##### Relationships & Data Flow
- Model Definitions ⇄ Training Engine
- Optimization Factory → Training Engine
- Orchestration Scripts → Training Engine & Visualization Engine
- Training Engine → Model Zoo (push to Hugging Face or local MODEL_ZOO)

##### Architectural Patterns & Principles
- Separation of concerns: distinct modules for data, model, training, optimization, orchestration, and docs.
- Pipeline pattern: clear pre-training pipeline vs. fine-tuning pipeline; both reuse data and model components but differ in decoder usage and classification head.
- Config-driven experiments: all hyperparameters and dataset settings flow from the orchestration scripts into engines and factories.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data source
    DataSource[(External Video Data)]:::external

    %% Data Ingestion & Preparation
    subgraph "Data Ingestion & Preparation"
        direction TB
        DATASET["DATASET.md"]:::data
        Datasets["datasets.py"]:::data
        Kinetics["kinetics.py"]:::data
        SSV2["ssv2.py"]:::data
    end

    %% Data Augmentation & Masking
    subgraph "Data Augmentation & Masking"
        direction TB
        Transforms["transforms.py"]:::data
        VideoTransforms["video_transforms.py"]:::data
        VolumeTransforms["volume_transforms.py"]:::data
        RandAugment["rand_augment.py"]:::data
        RandomErasing["random_erasing.py"]:::data
        Mixup["mixup.py"]:::data
        MaskingGen["masking_generator.py"]:::data
    end

    %% Model Definitions
    subgraph "Model Definitions"
        direction TB
        Encoder["Encoder (ViT)"]:::model
        Decoder["Decoder (Pre-training)"]:::model
        Classifier["Classification Head"]:::model
    end

    %% Training & Evaluation Engines
    subgraph "Training & Evaluation Engines"
        direction TB
        EnginePT["engine_for_pretraining.py"]:::engine
        EngineFT["engine_for_finetuning.py"]:::engine
    end

    %% Optimization & Config Factory
    OptimFactory["optim_factory.py"]:::optim

    %% Orchestration Scripts & Shell Helpers
    subgraph "Orchestration Scripts"
        direction TB
        RunPretrain["run_mae_pretraining.py"]:::script
        RunFinetune["run_class_finetuning.py"]:::script
        RunVis["run_videomae_vis.py"]:::script
        VisSh["vis.sh"]:::script
        ScriptsKinetics["scripts/kinetics/.../*.sh"]:::script
        ScriptsSSV2["scripts/ssv2/.../*.sh"]:::script
        ScriptsUCF["scripts/ucf101/.../*.sh"]:::script
    end

    %% Utilities & Logging
    Utils["utils.py"]:::util

    %% Checkpoints Storage
    Checkpoints[(Checkpoints Store)]:::storage

    %% Model Zoo Storage
    ModelZoo[(Model Zoo Storage)]:::storage

    %% Documentation & Model Zoo Docs
    subgraph "Documentation & Model Zoo"
        direction TB
        README["README.md"]:::doc
        PRETRAIN["PRETRAIN.md"]:::doc
        FINETUNE["FINETUNE.md"]:::doc
        INSTALL["INSTALL.md"]:::doc
        DATASET_DOC["DATASET.md"]:::doc
        MODEL_ZOO_DOC["MODEL_ZOO.md"]:::doc
    end

    %% External Services
    ExternalServices((Cloud Services)):::external

    %% Workflows
    DataSource --> DATASET
    DATASET --> Datasets
    Datasets --> Transforms
    Transforms --> Encoder

    Encoder --> Decoder
    Decoder --> EnginePT
    Encoder --> Classifier
    Classifier --> EngineFT

    EnginePT --> Checkpoints
    EngineFT --> Checkpoints
    EnginePT --> Utils
    EngineFT --> Utils
    EnginePT --> ModelZoo
    EngineFT --> ModelZoo

    OptimFactory -.-> EnginePT
    OptimFactory -.-> EngineFT

    RunPretrain -.-> EnginePT
    RunFinetune -.-> EngineFT
    RunVis -.-> Utils

    ExternalServices --> DataSource
    ExternalServices --> ModelZoo

    %% Click events
    click DATASET "https://github.com/mcg-nju/videomae/blob/main/DATASET.md"
    click Datasets "https://github.com/mcg-nju/videomae/blob/main/datasets.py"
    click Kinetics "https://github.com/mcg-nju/videomae/blob/main/kinetics.py"
    click SSV2 "https://github.com/mcg-nju/videomae/blob/main/ssv2.py"
    click Transforms "https://github.com/mcg-nju/videomae/blob/main/transforms.py"
    click VideoTransforms "https://github.com/mcg-nju/videomae/blob/main/video_transforms.py"
    click VolumeTransforms "https://github.com/mcg-nju/videomae/blob/main/volume_transforms.py"
    click RandAugment "https://github.com/mcg-nju/videomae/blob/main/rand_augment.py"
    click RandomErasing "https://github.com/mcg-nju/videomae/blob/main/random_erasing.py"
    click Mixup "https://github.com/mcg-nju/videomae/blob/main/mixup.py"
    click MaskingGen "https://github.com/mcg-nju/videomae/blob/main/masking_generator.py"
    click EnginePT "https://github.com/mcg-nju/videomae/blob/main/engine_for_pretraining.py"
    click EngineFT "https://github.com/mcg-nju/videomae/blob/main/engine_for_finetuning.py"
    click OptimFactory "https://github.com/mcg-nju/videomae/blob/main/optim_factory.py"
    click RunPretrain "https://github.com/mcg-nju/videomae/blob/main/run_mae_pretraining.py"
    click RunFinetune "https://github.com/mcg-nju/videomae/blob/main/run_class_finetuning.py"
    click RunVis "https://github.com/mcg-nju/videomae/blob/main/run_videomae_vis.py"
    click VisSh "https://github.com/mcg-nju/videomae/blob/main/vis.sh"
    click ScriptsKinetics "https://github.com/mcg-nju/videomae/blob/main/scripts/kinetics/.../*.sh"
    click ScriptsSSV2 "https://github.com/mcg-nju/videomae/blob/main/scripts/ssv2/.../*.sh"
    click ScriptsUCF "https://github.com/mcg-nju/videomae/blob/main/scripts/ucf101/.../*.sh"
    click Utils "https://github.com/mcg-nju/videomae/blob/main/utils.py"
    click README "https://github.com/mcg-nju/videomae/blob/main/README.md"
    click PRETRAIN "https://github.com/mcg-nju/videomae/blob/main/PRETRAIN.md"
    click FINETUNE "https://github.com/mcg-nju/videomae/blob/main/FINETUNE.md"
    click INSTALL "https://github.com/mcg-nju/videomae/blob/main/INSTALL.md"
    click DATASET_DOC "https://github.com/mcg-nju/videomae/blob/main/DATASET.md"
    click MODEL_ZOO_DOC "https://github.com/mcg-nju/videomae/blob/main/MODEL_ZOO.md"

    %% Styles
    classDef data fill:#a2d2ff,stroke:#000,color:#000
    classDef model fill:#b7e4c7,stroke:#000,color:#000
    classDef engine fill:#ffba08,stroke:#000,color:#000
    classDef optim fill:#ffd166,stroke:#000,color:#000
    classDef script fill:#ced4da,stroke:#000,color:#000
    classDef util fill:#f1f3f5,stroke:#000,color:#000
    classDef doc fill:#ffffff,stroke:#000,color:#000
    classDef external fill:#9d4edd,stroke:#000,color:#fff
    classDef storage fill:#ffc8dd,stroke:#000,color:#000
```
