# zebangcheng/emotion-llama architecture

> Architecture diagram of the GitHub repository zebangcheng/emotion-llama, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zebangcheng/emotion-llama
- Repository: https://github.com/zebangcheng/emotion-llama

## Overview

##### Project Type and Purpose
- The project is a full-stack research system focused on multimodal emotion recognition and reasoning.
- It processes multimodal inputs (audio, video, text) and uses several encoders (e.g., HuBERT for audio, EVA/MAE/VideoMAE for visual features) before fusing them via a modified LLaMA (instruction tuning applied to a large language model).
- It supports training, evaluation, and both a local and online demo (often via a Gradio interface) for real-time inference.

##### File Structure
- Top-level files (e.g., app.py, train.py, eval_emotion.py) indicate distinct entry points for demo, training, and evaluation.
- The “minigpt4” directory holds core framework modules:
– “models/” contains different model architectures: the base models, LLaMA integration, and MiniGPT-v2 enhancements.
– “datasets/” and “datasets/builders/” define how data is ingested and preprocessed.
– “processors/” shows how different modality-specific preprocessing (like blip_processors, randaugment) is performed.
– “runners/” and “tasks/” indicate the modular design where tasks (emotion recognition, reasoning) are handled in a decoupled manner.
- Evaluation and training configurations are found in “eval_configs” and “train_configs” respectively.
- Additional files (e.g., multiple README and markdown files) provide guidance on setup, dataset preparation (MERR), and even API usage (api_en.md, api_zh.md).

##### Key Components
a. Data Sources & Ingestion:
– Input Data: Video files and extracted audio (referenced in the “examples” folder and dataset configuration files).
– MERR dataset: Both coarse-grained and fine-grained annotations.
b. Preprocessing Components:
– Data Preprocessing & Feature Extraction Pipelines (within “minigpt4/datasets” and the instructions for feature extraction using external models such as HuBERT, EVA, MAE, VideoMAE).
– Dataset Builders: “minigpt4/datasets/builders” for organizing how raw data is transformed.
c. Model/Backbone:
– Multimodal Encoders: Separate modules for audio (HuBERT), global visual (EVA), local visual (MAE), and temporal (VideoMAE) encoders.
– Fusion & Shared Feature Space: Architecture that aligns these diverse modalities.
– Instruction Tuning: Modified LLaMA model integrated with MiniGPT-v2 (components under “minigpt4/models” and config files like minigpt_v2.yaml and Emotion-LLaMA_finetune.yaml).
d. Inference & Demo Interface:
– Demo Interface via “app.py” which launches a local Gradio-based webpage.
– API endpoints for local Gradio API (refer to api_en.md and api_zh.md).
e. Training and Evaluation Pipelines:
– Training scripts (train.py) with configuration files in “train_configs.”
– Evaluation setups (eval_emotion.py) using “eval_configs” to measure performance metrics (F1 scores, Clue Overlap, Label Overlap) on datasets such as MER2023 and MER2024.

##### Relationships and Interactions
– Data flows from the raw input (video, audio) → Preprocessing and feature extraction → Multimodal encoders → Fusion via instruction-tuned LLaMA → Output of emotion recognition and reasoning.
- Training pipeline feeds back into model updating while evaluation modules tap the trained model using checkpoints (noted in the configuration files).
- The demo interface accesses the inference pipeline to display results in real time.
- External services (like Hugging Face or Google Drive) provide pretrained weights and additional models.

##### Architectural Patterns and Key Technologies
- Modular design: Each modality and each processing step is encapsulated within its own module (directories such as “processors”, “datasets”, “models”).
- Pipeline organization: Well-separated stages for data ingestion, processing, model training, and inference.
- Usage of instruction tuning: Integration of prompts and multiple task-specific instruction pools (as defined in the dataset files) enables dynamic behavior based on tasks like “emotion” and “reason.”
- Technologies include Python, PyTorch (torchrun for distributed training), and Gradio for the demo UI.
- The configuration management is based on YAML files (e.g., environment.yaml, train_configs, eval_configs).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources & Ingestion
    subgraph "Data Sources & Ingestion"
        InData1["Input Video/Audio Data"]
        InData2["MERR Dataset"]
    end

    %% Preprocessing
    subgraph "Preprocessing"
        DS1["Dataset Ingestion"]
        DS2["Dataset Builders"]
        Proc1["Modality-specific Processors"]
    end

    %% Model/Backbone
    subgraph "Model/Backbone"
        Model1["Multimodal Encoders & Fusion"]
        Model2["LLaMA Integration"]
        Model3["MiniGPT-v2 Enhancements"]
    end

    %% Inference & Demo Interface
    subgraph "Inference & Demo Interface"
        Demo1["Demo Interface"]
        APIDocEN["API Documentation (EN)"]
        APIDocZH["API Documentation (ZH)"]
    end

    %% Training and Evaluation Pipelines
    subgraph "Training and Evaluation Pipelines"
        TrainScript["Training Script"]
        TrainConfig["Training Configurations"]
        EvalModule["Evaluation Module"]
        EvalConfig["Evaluation Configurations"]
    end

    %% Data Flow
    InData1 -->|"feeds"| DS1
    InData2 -->|"feeds"| DS1
    DS1 -->|"processes"| DS2
    DS2 -->|"builds"| Proc1
    Proc1 -->|"extracts features"| Model1
    Model1 -->|"integrates"| Model2
    Model1 -->|"enhances"| Model3
    Model2 -->|"outputs to"| Demo1
    Model3 -->|"outputs to"| Demo1
    Demo1 -->|"provides"| APIDocEN
    Demo1 -->|"provides"| APIDocZH

    %% Training and Evaluation Flow
    TrainScript -->|"updates"| Model1
    TrainConfig -->|"configures"| Model1
    Model1 -->|"evaluated by"| EvalModule
    EvalConfig -->|"configures"| EvalModule

    %% Class definitions for color coding
    classDef data fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef preprocess fill:#ABEBC6,stroke:#229954,stroke-width:2px;
    classDef model fill:#FAD7A0,stroke:#CA6F1E,stroke-width:2px;
    classDef ui fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
    classDef training fill:#D7BDE2,stroke:#8E44AD,stroke-width:2px;

    class InData1,InData2 data;
    class DS1,DS2,Proc1 preprocess;
    class Model1,Model2,Model3 model;
    class Demo1,APIDocEN,APIDocZH ui;
    class TrainScript,TrainConfig,EvalModule,EvalConfig training;

    %% Click Events
    click InData1 "https://github.com/zebangcheng/emotion-llama/tree/main//examples"
    click InData2 "https://github.com/zebangcheng/emotion-llama/tree/main//MERR"
    click DS1 "https://github.com/zebangcheng/emotion-llama/tree/main//minigpt4/datasets"
    click DS2 "https://github.com/zebangcheng/emotion-llama/tree/main//minigpt4/datasets/builders"
    click Proc1 "https://github.com/zebangcheng/emotion-llama/tree/main//minigpt4/processors"
    click Model1 "https://github.com/zebangcheng/emotion-llama/tree/main//minigpt4/models"
    click Model2 "https://github.com/zebangcheng/emotion-llama/blob/main//minigpt4/models/modeling_llama.py"
    click Model3 "https://github.com/zebangcheng/emotion-llama/blob/main//minigpt4/models/minigpt_v2.py"
    click Demo1 "https://github.com/zebangcheng/emotion-llama/blob/main//app.py"
    click APIDocEN "https://github.com/zebangcheng/emotion-llama/blob/main//api_en.md"
    click APIDocZH "https://github.com/zebangcheng/emotion-llama/blob/main//api_zh.md"
    click TrainScript "https://github.com/zebangcheng/emotion-llama/blob/main//train.py"
    click TrainConfig "https://github.com/zebangcheng/emotion-llama/tree/main//train_configs"
    click EvalModule "https://github.com/zebangcheng/emotion-llama/blob/main//eval_emotion.py"
    click EvalConfig "https://github.com/zebangcheng/emotion-llama/tree/main//eval_configs"
```
