How zebangcheng/emotion-llama works
Project Type and Purpose
• The project is a full-stack research system focused on multimodal emotion recognition and reasoning.
• It processes multimodal inputs (audio, video, text) and uses several encoders (e.g., HuBERT for audio, EVA/MAE/VideoMAE for visual features) before fusing them via a modified LLaMA (instruction tuning applied to a large language model).
• It supports training, evaluation, and both a local and online demo (often via a Gradio interface) for real-time inference.
File Structure
• Top-level files (e.g., app.py, train.py, eval_emotion.py) indicate distinct entry points for demo, training, and evaluation.
• The “minigpt4” directory holds core framework modules:
– “models/” contains different model architectures: the base models, LLaMA integration, and MiniGPT-v2 enhancements.
– “datasets/” and “datasets/builders/” define how data is ingested and preprocessed.
– “processors/” shows how different modality-specific preprocessing (like blip_processors, randaugment) is performed.
– “runners/” and “tasks/” indicate the modular design where tasks (emotion recognition, reasoning) are handled in a decoupled manner.
• Evaluation and training configurations are found in “eval_configs” and “train_configs” respectively.
• Additional files (e.g., multiple README and markdown files) provide guidance on setup, dataset preparation (MERR), and even API usage (api_en.md, api_zh.md).
Key Components
a. Data Sources & Ingestion:
– Input Data: Video files and extracted audio (referenced in the “examples” folder and dataset configuration files).
– MERR dataset: Both coarse-grained and fine-grained annotations.
b. Preprocessing Components:
– Data Preprocessing & Feature Extraction Pipelines (within “minigpt4/datasets” and the instructions for feature extraction using external models such as HuBERT, EVA, MAE, VideoMAE).
– Dataset Builders: “minigpt4/datasets/builders” for organizing how raw data is transformed.
c. Model/Backbone:
– Multimodal Encoders: Separate modules for audio (HuBERT), global visual (EVA), local visual (MAE), and temporal (VideoMAE) encoders.
– Fusion & Shared Feature Space: Architecture that aligns these diverse modalities.
– Instruction Tuning: Modified LLaMA model integrated with MiniGPT-v2 (components under “minigpt4/models” and config files like minigpt_v2.yaml and Emotion-LLaMA_finetune.yaml).
d. Inference & Demo Interface:
– Demo Interface via “app.py” which launches a local Gradio-based webpage.
– API endpoints for local Gradio API (refer to api_en.md and api_zh.md).
e. Training and Evaluation Pipelines:
– Training scripts (train.py) with configuration files in “train_configs.”
– Evaluation setups (eval_emotion.py) using “eval_configs” to measure performance metrics (F1 scores, Clue Overlap, Label Overlap) on datasets such as MER2023 and MER2024.
Relationships and Interactions
– Data flows from the raw input (video, audio) → Preprocessing and feature extraction → Multimodal encoders → Fusion via instruction-tuned LLaMA → Output of emotion recognition and reasoning.
• Training pipeline feeds back into model updating while evaluation modules tap the trained model using checkpoints (noted in the configuration files).
• The demo interface accesses the inference pipeline to display results in real time.
• External services (like Hugging Face or Google Drive) provide pretrained weights and additional models.
Architectural Patterns and Key Technologies
• Modular design: Each modality and each processing step is encapsulated within its own module (directories such as “processors”, “datasets”, “models”).
• Pipeline organization: Well-separated stages for data ingestion, processing, model training, and inference.
• Usage of instruction tuning: Integration of prompts and multiple task-specific instruction pools (as defined in the dataset files) enables dynamic behavior based on tasks like “emotion” and “reason.”
• Technologies include Python, PyTorch (torchrun for distributed training), and Gradio for the demo UI.
• The configuration management is based on YAML files (e.g., environment.yaml, train_configs, eval_configs).