How sunlicai/hicmae works
Project Type and Purpose
• This is a research‐oriented Python framework for self-supervised Audio-Visual Emotion Recognition (HiCMAE).
• It provides end-to-end pipelines for:
• Pre-training a Hierarchical Contrastive Masked Autoencoder on large unlabeled audio-visual data (e.g., VoxCeleb2)
• Fine-tuning on downstream labeled emotion datasets (CREMA-D, MAFW, etc.)
• Key goals: hierarchical masked reconstruction, cross-modal contrastive learning, multi-level feature fusion.
File-Structure Analysis
Top-level modules:
• preprocess/…
• Dataset-specific annotation generators (creamd_av.py, mafw_av.py, voxceleb2.py)
• Face and audio extractors (extract_face.py / extract_audio.py)
• transforms.py, video_transforms.py, volume_transforms.py, masking_generator.py
• Data-augmentation, masking, random erasing, MixUp
• datasets_av.py, kinetics_av.py, ssv2.py
• PyTorch Dataset classes for different data sources
• modeling_pretrain_av.py, modeling_finetune_av.py
• Encoder–decoder and classification/regression heads
• engine_for_pretraining_av.py, engine_for_finetuning_av.py, engine_for_finetuning_regression_av.py
• Training loops: loss calculation (masked autoencoding + contrastive), optimizer scheduling
• run_* and scripts/…
• Shell scripts and entry-points for launching experiments
• saved/…
• Preprocessed splits, intermediate CSVs, and model checkpoints
• utils.py, optim_factory.py, regression_metrics.py
• Helpers: learning rate schedulers, metric calculators
Architectural Insights from README
• Three-pronged hierarchical design:
a. Skip connections layer-to-layer between encoder and decoder
b. Contrastive losses applied at multiple intermediate encoder layers
c. Hierarchical feature fusion during fine-tuning
• Built atop VideoMAE & AudioMAE backbones (ViT-style transformers)
Components
a. Data Ingestion & Preprocessing
• Raw videos → extract_face.py → aligned frames
• Raw videos → extract_audio.py → 16 kHz waveforms
• Dataset-specific CSV generators
b. Data Augmentation & Masking
• video_transforms, volume_transforms, rand_augment, random_erasing, masking_generator
c. DataLoader / Dataset
• datasets_av.py and siblings load paired (video, audio, label) entries
d. Model Backbone
• VideoMAE-derived video encoder
• AudioMAE-derived audio encoder
e. Hierarchical Masked Autoencoder (HiCMAE)
• Encoder outputs at L1…Lk
• Skip connections feeding decoder for reconstruction
• Decoder reconstructs masked patches/modalities
f. Contrastive Learning Heads
• Projection heads at intermediate layers for audio-video alignment
g. Fine-tuning Heads
• Classification and regression heads (modeling_finetune_av.py)
• Hierarchical feature fusion block
h. Training Engines
• Pretrain engine: joint loss = reconstruction + multi-layer contrastive
• Finetune engine: cross-entropy (classification) or regression loss
i. Checkpointing & Logging
• TensorBoardX, saved/model, saved/data
j. External Tools & Dependencies
• PyTorch, torchvision, timm, einops, decord, mmcv, OpenFace, FFmpeg
Relationships & Data Flow
• Raw Data → Preprocessing → Augmentation/Masking → DataLoader
• DataLoader → Training Engine → Model Backbone → Heads → Loss → Optimizer
• Encoder intermediate outputs → Contrastive heads + Decoder (via skip connections)
• Training Engine → Checkpoint & Logs → saved/model
• Two modes: Pre-training and Fine-tuning with diverging heads but shared backbone.
Architectural Patterns & Principles
• Modular separation: data, transforms, model, engine, scripts
• Pipeline pattern: explicit dataflow from raw to final model
• Hierarchical design: multi-level supervision
• Extensibility: to add new datasets, augmentations, loss terms, backbones