How mcg-nju/videomae works
Project Type and Purpose
• This is an open-source PyTorch research library for self-supervised video pre-training (VideoMAE) and downstream fine-tuning.
• It implements a masked autoencoder on video sequences (plain ViT backbone) with extremely high masking ratios, supporting both pre-training and fine-tuning pipelines.
Top-Level Components
2.1 Data Ingestion & Preparation (Blue rectangle)
• DATASET.md, datasets.py, kinetics.py, ssv2.py
• Responsibilities: loading video datasets (Kinetics-400, Something-Something V2, UCF101), parsing metadata, producing raw video frames.
2.2 Data Augmentation & Masking (Blue rectangle)
• transforms.py, video_transforms.py, volume_transforms.py, rand_augment.py, random_erasing.py, mixup.py, masking_generator.py
• Responsibilities: spatial/temporal transforms, random erasing/mixup, tube masking generator for patch masking.
2.3 Model Definitions (Green rectangle)
• modeling_pretrain.py, modeling_finetune.py, functional.py
• Responsibilities:
• Encoder (ViT patch embedding, transformer blocks)
• Decoder (lightweight transformer for reconstruction in pre-training)
• Head for classification in fine-tuning
2.4 Training & Evaluation Engines (Orange rectangle)
• engine_for_pretraining.py, engine_for_finetuning.py
• Responsibilities: training loops, loss computation, backprop, metrics logging, checkpoint I/O.
2.5 Optimization & Config Factory (Yellow rectangle)
• optim_factory.py
• Responsibilities: optimizer and scheduler set-up (AdamW, weight decay, learning-rate schedules).
2.6 Orchestration Scripts (Gray rectangle)
• run_mae_pretraining.py, run_class_finetuning.py, run_videomae_vis.py, vis.sh
• Under scripts/… : dataset-specific shell scripts (pretrain.sh, finetune.sh) for Kinetics, SSV2, UCF101.
• Responsibilities: CLI interfaces, command-line argument parsing, setting up experiment folders, invoking engines with correct configs.
2.7 Utilities & Logging (Light-gray rectangle)
• utils.py
• Responsibilities: helper functions (checkpoint saving/loading, metric tracking, distributed training utilities).
• README.md, PRETRAIN.md, FINETUNE.md, INSTALL.md, DATASET.md, MODEL_ZOO.md
• Responsibilities: user instructions, dataset preparation, installation, links to pre-trained weights.
2.9 External Services (Purple cloud)
• Hugging Face Models/Spaces, PapersWithCode, external dataset links, license references.
Relationships & Data Flow
• Model Definitions ⇄ Training Engine
• Optimization Factory → Training Engine
• Orchestration Scripts → Training Engine & Visualization Engine
• Training Engine → Model Zoo (push to Hugging Face or local MODEL_ZOO)
Architectural Patterns & Principles
• Separation of concerns: distinct modules for data, model, training, optimization, orchestration, and docs.
• Pipeline pattern: clear pre-training pipeline vs. fine-tuning pipeline; both reuse data and model components but differ in decoder usage and classification head.
• Config-driven experiments: all hyperparameters and dataset settings flow from the orchestration scripts into engines and factories.