How warmcongee/sdumc works
Project Type and Purpose
• This is a research‐oriented multimodal sentiment analysis pipeline designed to train and evaluate a model that handles missing text modality.
• It consists of:
• A feature‐extraction layer (audio, text, visual)
• A dataset and data‐loading layer
• A modeling layer implementing self‐distillation and unified cross‐attention
• Training and inference scripts
• External tooling for feature extraction (OpenFace, FFmpeg, HuggingFace models)
Top-Level Components
1. Dataset Directory
• dataset/ holds raw and preprocessed labels (datasets_label/cmumosei-process/label_official.npz) and spaces for modality representations (features_mosei/*).
2. Feature Extraction Layer (feature_extraction/)
• audio: scripts to extract transformer embeddings (extract_transformers_embedding.py)
• text: HuggingFace text embedding extraction
• visual: OpenFace and MANet embeddings
• llm4wav: simulate text from audio using WavLM→Vicuna model
• Each subfolder has a run.sh orchestration script
3. Toolkit (toolkit/)
• data/: feature data structures (feat_data.py)
• dataloader/: dataset-specific loaders (CMU-MOSEI, IEMOCAP, etc.)
• models/: model definitions (UMCA, MIA, self-distillation, graph_mfn, transformers modules)
• preprocess/: dataset preprocessing routines
• utils/: loss functions (RNC), metrics, I/O utilities
4. Entrypoints
• Training: shell/main_text_missing_icassp.sh
• Inference: shell/main_text_missing_icassp_inference.sh
• Main validation scripts: main_frame_val_text_missing.py / _inference.py
5. Configuration & Dependencies
• requirements.txt, config.py for hyperparameters and paths
• External tools under tools/ (OpenFace, FFmpeg, HuggingFace checkpoints)
Component Interactions
• Data Flow:
1. Raw dataset → preprocessing (toolkit/preprocess) → saved in dataset/
2. Feature extraction scripts (feature_extraction/*) use external tools to generate embeddings → store in dataset/features_mosei/
3. The training script loads features via toolkit/data/feat_data.py and toolkit/dataloader/*
4. The model (UMCA + MIA + RNC) defined in toolkit/models consumes multimodal embeddings, handles missing-text case via llm4wav simulation
5. Loss and metrics computed in toolkit/utils → backprop → checkpoint saved
• Inference: same pipeline minus weight updates, produces evaluation metrics logged via metric.py
Architectural Patterns & Design Principles
• Modular pipeline: clear separation of feature extraction, data loading, model logic, and orchestration
• Plug-in extensibility: each modality extractor and dataloader lives in its own folder with standard interface (run.sh, Python script)
• Configuration-driven: config.py centralizes path and hyperparameter management
• Layered structure:
• Extraction layer (external tools & scripts)
• Data layer (I/O + preprocessing)
• Model layer (neural architectures + loss)
• Orchestration layer (shell scripts, main Python entrypoints)