# mkmas-guet/kuda architecture

> Architecture diagram of the GitHub repository mkmas-guet/kuda, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mkmas-guet/kuda
- Repository: https://github.com/mkmas-guet/kuda

## Overview

##### Project type and purpose
- This is an ML research codebase for “Knowledge-Guided Dynamic Modality Attention Fusion” in multimodal sentiment analysis (EMNLP 2024).
- It consists of two sequential stages:
a. Knowledge-Injection Pretraining (to adapt BERT to modality-specific knowledge)
b. Multimodal Sentiment-Analysis Training (to train the final fusion model on labeled datasets)

##### File structure
- Top‐level directories and files:
- core/ – reusable modules (dataset loader, metrics, scheduler, utilities)
- data/ – storage for the four MSA datasets (CH-SIMS, CH-SIMSv2, CMU-MOSI, CMU-MOSEI)
- models/ – model definitions (dynamic fusion, KIAdapter‐enhanced encoder, overall model wrapper)
- pretrainedModel/ – checkpoints for BERT and pretraining outputs
- log/ – runtime logs and evaluation reports
- opts.py – command‐line argument definitions
- pretrain.py – script for Stage 1 (knowledge inject pretraining)
- train.py – script for Stage 2 (final multimodal sentiment model training)

##### README for additional insights
- The pipeline uses HuggingFace BERT (Chinese and English)
- Emphasizes two distinct stages, each driven by its own Python entrypoint
- Notes a manual code patch in models/Encoder_KIAdapter.py to extract intermediate hidden states
- External dependencies: PyTorch 1.9.0, Python 3.8, NVIDIA GPU, plus dataset download links

##### Main components and their interactions
a. Data Layer
- Raw datasets in data/ (CH-SIMS, CH-SIMSv2, MOSI, MOSEI)
- data loaders in core/dataset.py
b. Core Library
- core/metric.py: performance metrics (e.g., accuracy, F1, regression metrics)
- core/scheduler.py: learning rate scheduling
- core/utils.py: I/O, logging helpers
c. Models
- Encoder_KIAdapter.py: modified Transformer encoder to output hidden states
- DyRoutFusion_CLS.py: dynamic routing fusion classifier
- OverallModal.py: orchestration of multimodal inputs → fusion → classifier
d. Pretrained Models
- pretrainedModel/BERT: HuggingFace checkpoints
- pretrainedModel/KnowledgeInjectPretraining: outputs of Stage 1
e. Scripts / Pipelines
- pretrain.py: reads data → loads BERT → applies KIAdapter → trains → saves weights
- train.py: reads data → loads pretrained weights → instantiates fusion model → trains → logs/evaluates
- opts.py: centralized argument parsing for both scripts
f. Logging and Outputs
- log/: stores training logs, checkpoints, evaluation results

##### Architectural patterns and design principles
- Modular separation of data, core logic, model definitions, and pipeline scripts
- Two‐stage pipeline, each with clearly defined inputs/outputs (pretraining → checkpoint → final training)
- Dependency injection via opts.py for hyperparameters, paths, and dataset selection
- Extensibility: new datasets or fusion modules can be plugged into the same core/model interfaces

##### Essential elements
- Clear component names (e.g., “core/dataset.py loader”)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Data Layer"
        Data["data/ (Raw Datasets)"]:::dataStorage
        Dataset["core/dataset.py\n(Data Loader)"]:::codeLib
    end

    subgraph "Core Library"
        Metric["core/metric.py\n(Metrics)"]:::codeLib
        Scheduler["core/scheduler.py\n(LR Scheduler)"]:::codeLib
        Utils["core/utils.py\n(Utilities)"]:::codeLib
    end

    subgraph "Models"
        Encoder["models/Encoder_KIAdapter.py\n(KI-Adapter Encoder)"]:::codeLib
        Fusion["models/DyRoutFusion_CLS.py\n(Dynamic Fusion Classifier)"]:::codeLib
        Wrapper["models/OverallModal.py\n(Modal Wrapper)"]:::codeLib
    end

    subgraph "Pretrained Models & Outputs"
        Pretrained["pretrainedModel/\n(Checkpoints & Stage-1 Outputs)"]:::dataStorage
    end

    subgraph "Pipelines"
        Opts["opts.py\n(Arg Parser)"]:::entry
        Pretrain["pretrain.py\n(Knowledge-Injection Pretraining)"]:::entry
        Train["train.py\n(Final Sentiment Training)"]:::entry
    end

    subgraph "Logging & Reports"
        Log["log/\n(Logs & Eval Results)"]:::dataStorage
    end

    subgraph "External Dependencies"
        HF["HuggingFace BERT"]:::external
        Torch["PyTorch 1.9.0"]:::external
        GPU["NVIDIA GPU"]:::external
        DLinks["Dataset Download Links"]:::external
    end

    Data -->|"raw data"| Dataset
    Dataset -->|"tokenized batches"| Pretrain
    Opts -->|"config params"| Pretrain
    HF --> Pretrain
    Pretrain -->|"pretrained weights"| Pretrained

    Pretrained -->|"weights"| Train
    Data -->|"tokenized batches"| Train
    Opts -->|"config params"| Train
    HF --> Train
    Train -->|"evaluation & checkpoints"| Log

    Pretrain -->|"uses"| Encoder
    Train -->|"uses"| Encoder
    Train -->|"uses"| Fusion
    Train -->|"uses"| Wrapper

    Pretrain -->|"uses"| Metric
    Pretrain -->|"uses"| Scheduler
    Pretrain -->|"uses"| Utils
    Train -->|"uses"| Metric
    Train -->|"uses"| Scheduler
    Train -->|"uses"| Utils

    click Data "https://github.com/mkmas-guet/kuda/tree/main/data/"
    click Dataset "https://github.com/mkmas-guet/kuda/blob/main/core/dataset.py"
    click Metric "https://github.com/mkmas-guet/kuda/blob/main/core/metric.py"
    click Scheduler "https://github.com/mkmas-guet/kuda/blob/main/core/scheduler.py"
    click Utils "https://github.com/mkmas-guet/kuda/blob/main/core/utils.py"
    click Encoder "https://github.com/mkmas-guet/kuda/blob/main/models/Encoder_KIAdapter.py"
    click Fusion "https://github.com/mkmas-guet/kuda/blob/main/models/DyRoutFusion_CLS.py"
    click Wrapper "https://github.com/mkmas-guet/kuda/blob/main/models/OverallModal.py"
    click Pretrained "https://github.com/mkmas-guet/kuda/tree/main/pretrainedModel/"
    click Opts "https://github.com/mkmas-guet/kuda/blob/main/opts.py"
    click Pretrain "https://github.com/mkmas-guet/kuda/blob/main/pretrain.py"
    click Train "https://github.com/mkmas-guet/kuda/blob/main/train.py"
    click Log "https://github.com/mkmas-guet/kuda/tree/main/log/"

    classDef dataStorage fill:#ffefc5,stroke:#eebf4f
    classDef codeLib fill:#e0f7fa,stroke:#4fc3f7
    classDef entry fill:#e8f5e9,stroke:#66bb6a
    classDef external fill:#f3e5f5,stroke:#ab47bc
```
