How opengvlab/videomaev2 works
Project Type and Purpose
• This is an open-source research codebase implementing “VideoMAE V2,” a video masked autoencoder for self-supervised pre-training and downstream fine-tuning on video action recognition and localization tasks.
• It provides data preparation, dual masking strategies, model definitions (encoder/decoder and classifier heads), training engines, feature extraction utilities, and shell scripts to run experiments at scale.
Top-Level Components
• dataset/ ← data ingestion & preprocessing
• models/ ← model architectures (pretrain & finetune)
• engine_for_pretraining.py
• engine_for_finetuning.py
• extract_tad_feature.py
• run_mae_pretraining.py
• run_class_finetuning.py
• scripts/ ← experiment wrappers (shell scripts for clusters)
• misc/ ← configuration maps and label files
• docs/ ← user manuals, dataset and training instructions
• optim_factory.py ← optimizer & scheduler factory
• utils.py ← shared utilities (logging, checkpointing, argument parsing)
Core Architectural Patterns
• Modular training engines (pretrain vs. finetune) driven by main entry points (run_*.py).
• Data-Pipeline abstraction via dataset/__init__.py and transforms, offering pluggable masking, augmentation, and loader strategies.
• Separation of concerns:
• Data preprocessing (dataset/)
• Model definition (models/)
• Training loop & optimization (engine_*.py + optim_factory.py)
• Experiment orchestration (scripts/ + run_*.py)
1. Data Storage & Preparation (blue)
• Raw video datasets on disk
• dataset/build.py & loader.py
• masking_generator.py (dual masking logic)
• transforms.py, video_transforms.py, rand_augment, random_erasing
2. Data Loader (blue)
• PyTorch DataLoader configured in dataset/__init__.py
3. Training Engines (orange)
• engine_for_pretraining.py
• engine_for_finetuning.py
4. Model Zoo (green)
• models/modeling_pretrain.py (ViT encoder + decoder)
• models/modeling_finetune.py (encoder + classification head)
5. Optimization & Logging (purple)
• optim_factory.py (LR schedulers, optimizers)
• utils.py (checkpoint, tensorboard, console logs)
6. Experiment Orchestration (grey)
• run_mae_pretraining.py
• run_class_finetuning.py
• scripts/pretrain/*.sh
• scripts/finetune/*.sh
7. Feature Extraction (teal)
• extract_tad_feature.py (for temporal action detection features)
1. Data Storage → Data Preparation → Data Loader → Training Engine
2. Training Engine ↔ Model Zoo (forward & backprop calls)
3. Training Engine ↔ Optimization & Logging (loss computation → optimizer updates → checkpoint & metrics logging)
4. Experiment Orchestration → Training Engine (CLI params, config files)
5. Feature Extraction ←– Data Loader + Pretrained Model → extract_tad_feature.py
Architectural Highlights & Patterns
• Dual-masking module sits early in the data pipeline, generating spatio-temporal masks for VideoMAE pretraining.
• Two distinct model flows:
• Pretrain: Autoencoder structure with encoder/decoder in modeling_pretrain.py
• Finetune: Encoder with classification head in modeling_finetune.py
• Optimizer factory abstracts optimizer & scheduler instantiation based on configuration.
• Shell scripts in scripts/ provide reproducible launch commands for different model scales and datasets.