# azyoung233/mse-adapter architecture

> Architecture diagram of the GitHub repository azyoung233/mse-adapter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/azyoung233/mse-adapter
- Repository: https://github.com/azyoung233/mse-adapter

## Overview

##### Project type and purpose
- This is a research‐oriented machine-learning toolkit (“MSE-Adapter”) that plugs into large-language models (LLMs) to perform multimodal sentiment analysis and emotion recognition.
- It provides implementations for three backbones: ChatGLM3-6B, Llama2-7B, and Qwen-1.8B, each in a separate subfolder.

##### File structure
Top level:
- MSE-ChatGLM3-6B/, MSE-Llama2-7B/, MSE-Qwen-1.8B/ — per-backbone codebases
- README.md, requirements.txt
Inside each backbone folder:
- config/ — training-mode and task-mode configurations (classification vs. regression)
- data/ — data loaders and preprocessors (DataPre.py, TextPre.py, getFeature, getLengths, load_data)
- models/
– AMIO.py — multimodal input/output adapter
– ChatGLM3 or core-model subpackage — LLM integration, tokenization, quantization
– multiTask/CMCM.py — multitask head (sentiment & emotion)
– subNets/Textmodel.py — auxiliary unimodal text encoder
- trains/ — training routines (ATIO.py for single-task, multiTask/CMCM.py)
- utils/ — shared helpers (functions.py, metricsTop.py)
- logs/ — run logs
- run.py — entrypoint for training and evaluation

##### README for additional insights
- Core workflow:
a. Specify dataset directory and backbone model path in run.py’s arguments
b. Data is loaded and preprocessed into features and lengths
c. MSE-Adapter (AMIO + subNets + multiTask) is wired into the frozen or quantized LLM
d. Training with 5 seeds; results persist under results/result
- External dependencies: Hugging Face datasets, virtual environment with fixed transformers version

##### Details
- MultiTask head coexisting with the adapter
- If space allows, note “config_classification.py” vs. “config_regression.py” as two branches of the Config Manager
- That only adapter parameters are trainable, while the backbone LLM remains frozen or quantized

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "External Storage"
        Datasets[(Datasets)]:::storage
        Checkpoints[(LLM Checkpoints)]:::storage
    end

    subgraph "Configuration Manager"
        ConfigClass["config_classification.py"]:::orchestration
        ConfigReg["config_regression.py"]:::orchestration
    end

    Run["run.py"]:::orchestration

    subgraph "Data Pipeline"
        Load["load_data.py"]:::data
        Raw["DataPre.py"]:::data
        TextP["TextPre.py"]:::data
        Feature["getFeature.py"]:::data
        Lengths["getLengths.py"]:::data
    end

    FeatureOut["Features & Lengths"]:::data

    subgraph "MSE-Adapter Plugin"
        AMIO["AMIO.py"]:::coreml
        TextModel["Textmodel.py"]:::coreml
        Head["CMCM.py"]:::coreml
    end

    Fused["Fused Representation"]:::coreml

    subgraph "LLM Core"
        Token["tokenization_chatglm.py"]:::coreml
        Model["modeling_chatglm.py"]:::coreml
        Quant["quantization.py"]:::coreml
    end

    Embeddings["Embeddings"]:::coreml
    Logits["Logits"]:::coreml

    subgraph "Training Engine"
        ATIO["ATIO.py"]:::coreml
        MTTrainer["CMCM.py"]:::coreml
    end

    Eval["metricsTop.py"]:::utils
    Utils["functions.py"]:::utils

    Logs[(Logs/Results)]:::storage

    %% Orchestration
    ConfigClass --> Run
    ConfigReg --> Run

    %% Data flow
    Datasets --> Load --> Raw --> TextP --> Feature --> Lengths --> FeatureOut
    Checkpoints --> Token --> Model --> Embeddings

    Run --> Load
    Run --> Token
    Run --> ATIO
    Run --> MTTrainer

    FeatureOut --> AMIO
    Embeddings --> AMIO
    AMIO --> Fused
    TextModel --> Fused
    Head --> Fused

    Fused --> Model
    Model --> Logits

    Logits --> ATIO
    Logits --> MTTrainer

    ATIO --> Logs
    MTTrainer --> Logs

    ATIO --> Eval
    MTTrainer --> Eval
    Eval --> Logs

    Utils --> Run

    %% Click events
    click ConfigClass "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/config/config_classification.py"
    click ConfigReg "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/config/config_regression.py"
    click Load "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/data/load_data.py"
    click Raw "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/data/DataPre.py"
    click TextP "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/data/TextPre.py"
    click Feature "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/data/getFeature.py"
    click Lengths "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/data/getLengths.py"
    click AMIO "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/models/AMIO.py"
    click TextModel "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/models/subNets/Textmodel.py"
    click Head "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/models/multiTask/CMCM.py"
    click Token "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/models/ChatGLM3/tokenization_chatglm.py"
    click Model "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/models/ChatGLM3/modeling_chatglm.py"
    click Quant "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/models/ChatGLM3/quantization.py"
    click ATIO "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/trains/ATIO.py"
    click MTTrainer "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/trains/multiTask/CMCM.py"
    click Eval "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/utils/metricsTop.py"
    click Utils "https://github.com/azyoung233/mse-adapter/blob/main/MSE-ChatGLM3-6B/utils/functions.py"
    click Logs "https://github.com/azyoung233/mse-adapter/tree/main/MSE-ChatGLM3-6B/logs/"

    %% Styles
    classDef coreml fill:#ADD8E6,stroke:#000,color:#000
    classDef data fill:#98FB98,stroke:#000,color:#000
    classDef orchestration fill:#FFA500,stroke:#000,color:#000
    classDef utils fill:#D3D3D3,stroke:#000,color:#000
    classDef storage fill:#FFD700,stroke:#000,color:#000
```
