How wooyoohl/mer2024-semi works
Main Components
• Data Input & Preprocessing: Even though the repository does not directly include data preprocessing scripts, the README and related links mention feature extraction for utterance-level and frame-level features. Represent an input layer that provides pre-extracted features or raw multimodal data.
• Pseudo-Labeling Mechanism: The README discusses the pseudo-label threshold configuration. This component is responsible for assigning reliable labels based on the model’s confidence for emotion recognition.
• Models Module: This is the heart of the project. Within the models directory, there are:
– Contrastive_model.py: Contains implementations for contrastive learning between modalities, between original and noisy features, and the computation of contrastive loss.
– Noise_scheduler.py: Implements noisy embedding construction, which adds perturbations to the features.
– Pretrain_contras.py: Likely sets up or runs a pretraining phase for contrastive learning.
• External Dependencies & Environment: Note the dependencies (PyTorch, numpy, etc.) provided in the README and indicate that the system runs within a Conda environment, connecting the model’s functionality with the external tools.
Relationships and Interactions
• Loss Calculation: Show how the output of the contrastive comparisons feeds into a loss computation module within the Contrastive_model.
• Training Pipeline: Connect all these modules with a central “Training Engine” that iterates over data batches, applies data augmentation, computes pseudo-labels, and updates the model via contrastive loss.
Architectural Patterns and Design Principles
• Layered Modular Structure: Emphasize that the functionally distinct components—data input, noise scheduling, contrastive-based learning, and pseudo-labeling—are modularized. This reflects separation of concerns.
• Self-training and Contrastive Paradigm: Highlight that the system exploits self-training combined with contrastive learning modules, which is a novel aspect of the design.
Relevant Technologies
• PyTorch: Mark it next to the modules responsible for model operations.
• Conda Environment: Mention that the system is compiled and executed in a Python 3.9 environment with a suite of scientific libraries.
Final Assembly
• Next, branch into pseudo-labeling (with threshold controls) and noise injection.
• Feed these into the “Contrastive Model Module”, which itself has sub-modules for modality contrast and original-noisy feature contrast.
• Merge the outputs downstream into a “Loss Calculation” component, which then informs the “Training Engine”.
• Finally, note external integrations with the MER2024 baseline and dependency environment.