# ricfrr/mpuc-mer architecture

> Architecture diagram of the GitHub repository ricfrr/mpuc-mer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ricfrr/mpuc-mer
- Repository: https://github.com/ricfrr/mpuc-mer

## Overview

##### Project Type and Purpose
- This is a PyTorch‐based machine learning research codebase implementing an unsupervised multimodal emotion recognition pipeline.
- It ingests text, audio, and video data (from CMU-MOSEI or RAVDESS), learns modality-pairwise features via contrastive loss, then fuses features at inference to classify discrete emotions.

##### Top‐Level Components
- Configuration (config/)
- YAML files defining dataset paths, modalities enabled, model hyperparameters, training schedule, wandb logging, etc.
- Adjacency matrix (*.npy) for graph-based modality.
- Data Ingestion (data/)
- CMU_MOSEI.py and RAVDESS.py: dataset classes, frame extraction, mel-spectrogram or graph adjacency preparation.
- Core Entry Points
- main.py: parses CLI args (config, device, dataset, wandb), instantiates components, kicks off training or evaluation.
- train.py: orchestrates training loop with unsupervised contrastive loss, scheduler, checkpointing.
- Models (models/)
- AudioEncoder.py, VideoEncoder.py, Encoder.py (text), STGCN.py (graph convolution), TCN.py (temporal conv), attention.py for intra-modality attention, LinearClassifier.py for downstream classification.
- Utilities (utils/)
- Loss.py: unsupervised contrastive loss functions.
- SupConSeparate.py: pairs modalities for contrastive training.
- ModelMonitor.py: checkpointing, early stopping, logging.
- evaluate.py: downstream emotion classifier evaluation.
- encoder_train.py: helper to pretrain encoders.
- utils.py: shared helpers (IO, transforms).

##### Data Flow and Relationships
- CONFIG → DATA LOADERS → PREPROCESSING
· Paths, n_frames, modality flags, adjacency matrix
- DATA → ENCODERS (text/audio/video/graph)
- ENCODED FEATURES → PAIRWISE CONTRASTIVE LOSS (via SupConSeparate & Loss)
- TRAINING LOOP (train.py) → UPDATE MODEL PARAMETERS
- INFERENCE: EXTRACT f_t, f_a, f_v, f_l PRE-PROJECTION → CONCAT → LinearClassifier
- MODEL MONITORING & LOGGING → WANDB

##### Architectural Patterns & Design Principles
- Modular separation: config, data, models, utils
- Pluggable encoders—switch backbone via config (e.g., STGCN vs TCN)
- Single-process monolith with clear layers rather than microservices
- Late fusion design: unsupervised training per modality pair, supervised classification only at inference
- Dependency injection: config‐driven instantiation of datasets, models, optimizers

## Diagram (Mermaid)

```mermaid
graph TB
    %% Top-level orchestration
    Main["main.py"]:::core
    Train["train.py"]:::core

    %% Configuration
    subgraph "Configuration" 
        direction TB
        ParamsMOSEI["params_MOSEI.yaml"]:::data
        ParamsRAVDESS["params_ravdess.yaml"]:::data
        Adj["adj.npy"]:::data
    end

    %% Data Sources & Ingestion
    subgraph "Data Ingestion" 
        direction TB
        DS1["CMU-MOSEI Dataset"]:::data
        DS2["RAVDESS Dataset"]:::data
        Loader["Data Loader & Preprocessing"]:::data
    end

    %% Encoders
    subgraph "Encoders (Pluggable)" 
        direction TB
        TextEnc["Text Encoder"]:::model
        AudioEnc["Audio Encoder"]:::model
        VideoEnc["Video Encoder"]:::model
        GraphEnc["Graph Encoder (STGCN)"]:::model
        TCNEnc["Temporal Conv (TCN)"]:::model
        Attention["Intra-Modality Attention"]:::model
    end

    %% Contrastive Training Utilities
    subgraph "Contrastive Trainer" 
        direction TB
        SupCon["SupConSeparate"]:::util
        LossFn["Loss Functions"]:::util
    end

    %% Inference & Classification
    Fusion["Inference Fusion + Classifier"]:::model
    Output["Predictions"]:::model

    %% Monitoring & Logging
    Monitor["ModelMonitor"]:::util
    WandB(>WandB<):::ext

    %% Relationships
    Main -->|"uses"| ParamsMOSEI
    Main -->|"uses"| ParamsRAVDESS
    Main -->|"uses"| Adj
    Main -->|"loads"| DS1
    Main -->|"loads"| DS2
    DS1 --> Loader
    DS2 --> Loader
    Loader --> TextEnc
    Loader --> AudioEnc
    Loader --> VideoEnc
    Loader --> GraphEnc
    Loader --> TCNEnc
    TextEnc --> SupCon
    AudioEnc --> SupCon
    VideoEnc --> SupCon
    GraphEnc --> SupCon
    TCNEnc --> SupCon
    SupCon --> LossFn
    LossFn --> Train
    Train -->|"updates"| TextEnc
    Train -->|"updates"| AudioEnc
    Train -->|"updates"| VideoEnc
    Train -->|"updates"| GraphEnc
    Train -->|"updates"| TCNEnc
    Train --> Monitor
    Monitor --> WandB
    TextEnc --> Fusion
    AudioEnc --> Fusion
    VideoEnc --> Fusion
    GraphEnc --> Fusion
    TCNEnc --> Fusion
    Fusion --> Output

    %% Click Events
    click ParamsMOSEI "https://github.com/ricfrr/mpuc-mer/blob/main/config/params_MOSEI.yaml"
    click ParamsRAVDESS "https://github.com/ricfrr/mpuc-mer/blob/main/config/params_ravdess.yaml"
    click Adj "https://github.com/ricfrr/mpuc-mer/blob/main/config/adj.npy"
    click DS1 "https://github.com/ricfrr/mpuc-mer/blob/main/data/CMU_MOSEI.py"
    click DS2 "https://github.com/ricfrr/mpuc-mer/blob/main/data/RAVDESS.py"
    click Main "https://github.com/ricfrr/mpuc-mer/blob/main/main.py"
    click Train "https://github.com/ricfrr/mpuc-mer/blob/main/train.py"
    click AudioEnc "https://github.com/ricfrr/mpuc-mer/blob/main/models/AudioEncoder.py"
    click TextEnc "https://github.com/ricfrr/mpuc-mer/blob/main/models/Encoder.py"
    click VideoEnc "https://github.com/ricfrr/mpuc-mer/blob/main/models/VideoEncoder.py"
    click GraphEnc "https://github.com/ricfrr/mpuc-mer/blob/main/models/STGCN.py"
    click TCNEnc "https://github.com/ricfrr/mpuc-mer/blob/main/models/TCN.py"
    click Attention "https://github.com/ricfrr/mpuc-mer/blob/main/models/attention.py"
    click Fusion "https://github.com/ricfrr/mpuc-mer/blob/main/models/LinearClassifier.py"
    click SupCon "https://github.com/ricfrr/mpuc-mer/blob/main/utils/SupConSeparate.py"
    click LossFn "https://github.com/ricfrr/mpuc-mer/blob/main/utils/Loss.py"
    click Monitor "https://github.com/ricfrr/mpuc-mer/blob/main/utils/ModelMonitor.py"

    %% Styles
    classDef data fill:#e0ffe0,stroke:#080;
    classDef model fill:#e0ecff,stroke:#005;
    classDef util fill:#f0f0f0,stroke:#666;
    classDef ext fill:#ffe0b3,stroke:#e65c00;
    classDef core fill:#f5faff,stroke:#3399ff,stroke-width:1.5px;
```
