How mkmas-guet/kuda works
Project type and purpose
• This is an ML research codebase for “Knowledge-Guided Dynamic Modality Attention Fusion” in multimodal sentiment analysis (EMNLP 2024).
• It consists of two sequential stages:
a. Knowledge-Injection Pretraining (to adapt BERT to modality-specific knowledge)
b. Multimodal Sentiment-Analysis Training (to train the final fusion model on labeled datasets)
File structure
• Top‐level directories and files:
• core/ – reusable modules (dataset loader, metrics, scheduler, utilities)
• data/ – storage for the four MSA datasets (CH-SIMS, CH-SIMSv2, CMU-MOSI, CMU-MOSEI)
• models/ – model definitions (dynamic fusion, KIAdapter‐enhanced encoder, overall model wrapper)
• pretrainedModel/ – checkpoints for BERT and pretraining outputs
• log/ – runtime logs and evaluation reports
• opts.py – command‐line argument definitions
• pretrain.py – script for Stage 1 (knowledge inject pretraining)
• train.py – script for Stage 2 (final multimodal sentiment model training)
README for additional insights
• The pipeline uses HuggingFace BERT (Chinese and English)
• Emphasizes two distinct stages, each driven by its own Python entrypoint
• Notes a manual code patch in models/Encoder_KIAdapter.py to extract intermediate hidden states
• External dependencies: PyTorch 1.9.0, Python 3.8, NVIDIA GPU, plus dataset download links
Main components and their interactions
a. Data Layer
• Raw datasets in data/ (CH-SIMS, CH-SIMSv2, MOSI, MOSEI)
• data loaders in core/dataset.py
b. Core Library
• core/metric.py: performance metrics (e.g., accuracy, F1, regression metrics)
• core/scheduler.py: learning rate scheduling
• core/utils.py: I/O, logging helpers
c. Models
• Encoder_KIAdapter.py: modified Transformer encoder to output hidden states
• DyRoutFusion_CLS.py: dynamic routing fusion classifier
• OverallModal.py: orchestration of multimodal inputs → fusion → classifier
d. Pretrained Models
• pretrainedModel/BERT: HuggingFace checkpoints
• pretrainedModel/KnowledgeInjectPretraining: outputs of Stage 1
e. Scripts / Pipelines
• pretrain.py: reads data → loads BERT → applies KIAdapter → trains → saves weights
• train.py: reads data → loads pretrained weights → instantiates fusion model → trains → logs/evaluates
• opts.py: centralized argument parsing for both scripts
f. Logging and Outputs
• log/: stores training logs, checkpoints, evaluation results
Architectural patterns and design principles
• Modular separation of data, core logic, model definitions, and pipeline scripts
• Two‐stage pipeline, each with clearly defined inputs/outputs (pretraining → checkpoint → final training)
• Dependency injection via opts.py for hyperparameters, paths, and dataset selection
• Extensibility: new datasets or fusion modules can be plugged into the same core/model interfaces
Essential elements
• Clear component names (e.g., “core/dataset.py loader”)