# warmcongee/sdumc architecture

> Architecture diagram of the GitHub repository warmcongee/sdumc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/warmcongee/sdumc
- Repository: https://github.com/warmcongee/sdumc

## Overview

##### Project Type and Purpose
- This is a research‐oriented multimodal sentiment analysis pipeline designed to train and evaluate a model that handles missing text modality.
- It consists of:
- A feature‐extraction layer (audio, text, visual)
- A dataset and data‐loading layer
- A modeling layer implementing self‐distillation and unified cross‐attention
- Training and inference scripts
- External tooling for feature extraction (OpenFace, FFmpeg, HuggingFace models)

##### Top-Level Components
1. Dataset Directory
- `dataset/` holds raw and preprocessed labels (`datasets_label/cmumosei-process/label_official.npz`) and spaces for modality representations (features_mosei/*).
2. Feature Extraction Layer (`feature_extraction/`)
- audio: scripts to extract transformer embeddings (`extract_transformers_embedding.py`)
- text: HuggingFace text embedding extraction
- visual: OpenFace and MANet embeddings
- llm4wav: simulate text from audio using WavLM→Vicuna model
- Each subfolder has a `run.sh` orchestration script
3. Toolkit (`toolkit/`)
- `data/`: feature data structures (`feat_data.py`)
- `dataloader/`: dataset-specific loaders (CMU-MOSEI, IEMOCAP, etc.)
- `models/`: model definitions (UMCA, MIA, self-distillation, graph_mfn, transformers modules)
- `preprocess/`: dataset preprocessing routines
- `utils/`: loss functions (RNC), metrics, I/O utilities
4. Entrypoints
- Training: `shell/main_text_missing_icassp.sh`
- Inference: `shell/main_text_missing_icassp_inference.sh`
- Main validation scripts: `main_frame_val_text_missing.py` / `_inference.py`
5. Configuration & Dependencies
- `requirements.txt`, `config.py` for hyperparameters and paths
- External tools under `tools/` (OpenFace, FFmpeg, HuggingFace checkpoints)

##### Component Interactions
- Data Flow:
1. Raw dataset → preprocessing (`toolkit/preprocess`) → saved in `dataset/`
2. Feature extraction scripts (`feature_extraction/*`) use external tools to generate embeddings → store in `dataset/features_mosei/`
3. The training script loads features via `toolkit/data/feat_data.py` and `toolkit/dataloader/*`
4. The model (UMCA + MIA + RNC) defined in `toolkit/models` consumes multimodal embeddings, handles missing-text case via llm4wav simulation
5. Loss and metrics computed in `toolkit/utils` → backprop → checkpoint saved
- Inference: same pipeline minus weight updates, produces evaluation metrics logged via `metric.py`

##### Architectural Patterns & Design Principles
- Modular pipeline: clear separation of feature extraction, data loading, model logic, and orchestration
- Plug-in extensibility: each modality extractor and dataloader lives in its own folder with standard interface (`run.sh`, Python script)
- Configuration-driven: `config.py` centralizes path and hyperparameter management
- Layered structure:
- Extraction layer (external tools & scripts)
- Data layer (I/O + preprocessing)
- Model layer (neural architectures + loss)
- Orchestration layer (shell scripts, main Python entrypoints)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Tools
    subgraph "External Tools"
        direction TB
        OpenFace["OpenFace"]:::external
        FFmpeg["FFmpeg"]:::external
        HF["HuggingFace Checkpoints"]:::external
        WavLMVicuna["WavLM→Vicuna"]:::external
    end

    %% Feature Extraction Layer
    subgraph "Feature Extraction Layer"
        direction TB
        Audio["Audio Extraction"]:::code
        TextFE["Text Extraction"]:::code
        Visual["Visual Extraction"]:::code
        LLM4wav["LLM4wav Simulation"]:::code
    end

    %% Dataset Storage
    subgraph "Dataset Storage"
        direction TB
        Dataset["dataset/ (Raw & Preprocessed)"]:::storage
    end

    %% Data Loading & Preprocessing
    subgraph "Data Layer"
        direction TB
        Preprocess["Preprocessing (toolkit/preprocess)"]:::code
        Dataloaders["Dataloaders (toolkit/dataloader)"]:::code
        FeatData["Feature Data Structs (toolkit/data)"]:::code
    end

    %% Model Components
    subgraph "Model Layer"
        direction TB
        UMCA["UMCA (wengnet_mosei_mult_views_text_missing)"]:::model
        DST["Missing-Text Attention (dst_att)"]:::model
        MIA["MIA Component (misa)"]:::model
        MFN["Graph MFN Baseline (graph_mfn)"]:::model
        Transformer["Transformer Modules"]:::model
    end

    %% Utils
    subgraph "Utils"
        direction TB
        Loss["Loss Functions (RNC)"]:::code
        Metrics["Metrics Utils"]:::code
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        TrainShell["Training Shell"]:::orchestrator
        InferShell["Inference Shell"]:::orchestrator
        ValEntry["Validation Script"]:::orchestrator
        InferEntry["Inference Script"]:::orchestrator
        Config["config.py / requirements.txt"]:::orchestrator
    end

    %% Outputs
    subgraph "Outputs"
        direction TB
        Checkpoints["Model Checkpoints (.pt)"]:::storage
        EvalMetrics["Evaluation Metrics"]:::storage
    end

    %% Connections
    OpenFace -->|used by| Audio
    FFmpeg -->|used by| Audio
    HF -->|used by| TextFE
    OpenFace -->|used by| Visual
    HF -->|used by| Visual
    WavLMVicuna -->|used by| LLM4wav

    Audio -->|embeddings| Dataset
    TextFE -->|embeddings| Dataset
    Visual -->|embeddings| Dataset
    LLM4wav -->|simulated text| Dataset

    Dataset -->|data files| Preprocess
    Preprocess -->|cleaned data| Dataloaders
    Dataset -->|features| FeatData
    FeatData -->|tensors| Dataloaders

    Dataloaders -->|batches| UMCA
    Dataloaders -->|batches| DST
    Dataloaders -->|batches| MIA
    Dataloaders -->|batches| MFN
    Dataloaders -->|batches| Transformer

    UMCA -->|calls| DST
    UMCA -->|calls| MIA
    UMCA -->|calls| Transformer

    UMCA -->|weights| Checkpoints
    MFN -->|weights| Checkpoints

    UMCA -->|outputs| Metrics
    DST -->|outputs| Metrics
    MIA -->|outputs| Metrics
    MFN -->|outputs| Metrics

    Metrics -->|loss values| Loss
    Loss -->|backprop| UMCA

    TrainShell -->|runs| Preprocess
    TrainShell -->|runs| Audio
    TrainShell -->|runs| TextFE
    TrainShell -->|runs| Visual
    TrainShell -->|runs| LLM4wav
    TrainShell -->|runs| Dataloaders
    TrainShell -->|runs| UMCA

    InferShell -->|runs| Dataloaders
    InferShell -->|loads| Checkpoints
    InferShell -->|runs| UMCA
    InferShell -->|runs| Metrics
    InferShell -->|outputs| EvalMetrics

    ValEntry -->|validates| UMCA
    InferEntry -->|infers| UMCA

    Config -->|configures| TrainShell
    Config -->|configures| InferShell

    %% Click Events
    click Audio "https://github.com/warmcongee/sdumc/blob/main/feature_extraction/audio/extract_transformers_embedding.py"
    click Audio "https://github.com/warmcongee/sdumc/blob/main/feature_extraction/audio/run.sh"
    click TextFE "https://github.com/warmcongee/sdumc/blob/main/feature_extraction/text/extract_text_embedding_huggingface.py"
    click TextFE "https://github.com/warmcongee/sdumc/blob/main/feature_extraction/text/run.sh"
    click Visual "https://github.com/warmcongee/sdumc/blob/main/feature_extraction/visual/extract_openface.py"
    click Visual "https://github.com/warmcongee/sdumc/blob/main/feature_extraction/visual/extract_manet_embedding.py"
    click Visual "https://github.com/warmcongee/sdumc/blob/main/feature_extraction/visual/run_visual.sh"
    click LLM4wav "https://github.com/warmcongee/sdumc/blob/main/feature_extraction/llm4wav/extract_wavlm_vicuna.py"
    click LLM4wav "https://github.com/warmcongee/sdumc/blob/main/feature_extraction/llm4wav/run.sh"
    click Dataset "https://github.com/warmcongee/sdumc/tree/main/dataset/"
    click FeatData "https://github.com/warmcongee/sdumc/blob/main/toolkit/data/feat_data.py"
    click Dataloaders "https://github.com/warmcongee/sdumc/tree/main/toolkit/dataloader/"
    click Preprocess "https://github.com/warmcongee/sdumc/blob/main/toolkit/preprocess/cmumosei.py"
    click Preprocess "https://github.com/warmcongee/sdumc/blob/main/toolkit/preprocess/cmumosi.py"
    click UMCA "https://github.com/warmcongee/sdumc/blob/main/toolkit/models/wengnet_mosei_mult_views_text_missing.py"
    click DST "https://github.com/warmcongee/sdumc/blob/main/toolkit/models/dst_att.py"
    click MIA "https://github.com/warmcongee/sdumc/blob/main/toolkit/models/misa.py"
    click MFN "https://github.com/warmcongee/sdumc/blob/main/toolkit/models/graph_mfn.py"
    click Transformer "https://github.com/warmcongee/sdumc/blob/main/toolkit/models/modules/transformers_encoder/transformer.py"
    click Loss "https://github.com/warmcongee/sdumc/blob/main/toolkit/utils/loss.py"
    click Metrics "https://github.com/warmcongee/sdumc/blob/main/toolkit/utils/metric.py"
    click Metrics "https://github.com/warmcongee/sdumc/blob/main/metric.py"
    click TrainShell "https://github.com/warmcongee/sdumc/blob/main/shell/main_text_missing_icassp.sh"
    click InferShell "https://github.com/warmcongee/sdumc/blob/main/shell/main_text_missing_icassp_inference.sh"
    click ValEntry "https://github.com/warmcongee/sdumc/blob/main/main_frame_val_text_missing.py"
    click InferEntry "https://github.com/warmcongee/sdumc/blob/main/main_frame_val_text_missing_inference.py"
    click Config "https://github.com/warmcongee/sdumc/blob/main/config.py"
    click Config "https://github.com/warmcongee/sdumc/blob/main/requirements.txt"

    %% Styles
    classDef external fill:#A5D6A7,stroke:#2E7D32;
    classDef code fill:#FFCC80,stroke:#EF6C00;
    classDef storage fill:#90CAF9,stroke:#1565C0;
    classDef orchestrator fill:#CE93D8,stroke:#6A1B9A;
    classDef model fill:#FFE082,stroke:#F57C00;
```
