How ricfrr/mpuc-mer works
Project Type and Purpose
• This is a PyTorch‐based machine learning research codebase implementing an unsupervised multimodal emotion recognition pipeline.
• It ingests text, audio, and video data (from CMU-MOSEI or RAVDESS), learns modality-pairwise features via contrastive loss, then fuses features at inference to classify discrete emotions.
Top‐Level Components
• Configuration (config/)
• YAML files defining dataset paths, modalities enabled, model hyperparameters, training schedule, wandb logging, etc.
• Adjacency matrix (*.npy) for graph-based modality.
• Data Ingestion (data/)
• CMU_MOSEI.py and RAVDESS.py: dataset classes, frame extraction, mel-spectrogram or graph adjacency preparation.
• Core Entry Points
• main.py: parses CLI args (config, device, dataset, wandb), instantiates components, kicks off training or evaluation.
• train.py: orchestrates training loop with unsupervised contrastive loss, scheduler, checkpointing.
• Models (models/)
• AudioEncoder.py, VideoEncoder.py, Encoder.py (text), STGCN.py (graph convolution), TCN.py (temporal conv), attention.py for intra-modality attention, LinearClassifier.py for downstream classification.
• Utilities (utils/)
• Loss.py: unsupervised contrastive loss functions.
• SupConSeparate.py: pairs modalities for contrastive training.
• ModelMonitor.py: checkpointing, early stopping, logging.
• evaluate.py: downstream emotion classifier evaluation.
• encoder_train.py: helper to pretrain encoders.
• utils.py: shared helpers (IO, transforms).
Data Flow and Relationships
• CONFIG → DATA LOADERS → PREPROCESSING
· Paths, n_frames, modality flags, adjacency matrix
• DATA → ENCODERS (text/audio/video/graph)
• ENCODED FEATURES → PAIRWISE CONTRASTIVE LOSS (via SupConSeparate & Loss)
• TRAINING LOOP (train.py) → UPDATE MODEL PARAMETERS
• INFERENCE: EXTRACT f_t, f_a, f_v, f_l PRE-PROJECTION → CONCAT → LinearClassifier
• MODEL MONITORING & LOGGING → WANDB
Architectural Patterns & Design Principles
• Modular separation: config, data, models, utils
• Pluggable encoders—switch backbone via config (e.g., STGCN vs TCN)
• Single-process monolith with clear layers rather than microservices
• Late fusion design: unsupervised training per modality pair, supervised classification only at inference
• Dependency injection: config‐driven instantiation of datasets, models, optimizers