# maelfabien/multimodal-emotion-recognition architecture

> Architecture diagram of the GitHub repository maelfabien/multimodal-emotion-recognition, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/maelfabien/multimodal-emotion-recognition
- Repository: https://github.com/maelfabien/multimodal-emotion-recognition

## Overview

##### Main Components
- Text Analysis Module – Handles natural language processing (NLP), including custom text pre-processing (tokenization, cleaning, lemmatization, etc.), Word2Vec embedding, and a hybrid CNN-LSTM based neural network for personality trait prediction.
- Audio Analysis Module – Manages speech emotion recognition by processing voice recordings through audio signal discretization, log-mel-spectrogram extraction, and a deep learning architecture (using a Time Distributed CNN followed by LSTM layers) as well as an alternative SVM-based approach.
- Video Analysis Module – Involves video-based emotion detection. It first processes video input through face detection (using HOG features and facial landmark detection), resizes the face images and then applies an XCeption-based CNN model to recognize emotions.
- Web Application (Flask) – Orchestrates the interaction between the user and the backend. It integrates the outputs of text, audio, and video pipelines, displays results through its HTML templates, and may trigger additional data visualization (with D3.js and other JS libraries).
- (Planned) Ensemble Model – Although not implemented in the current release, it is intended to combine decisions from text, audio, and video processing pipelines.

- Input Layer(s): Show separate entry points for text, audio, and video data.
- Pre-processing Pipelines: For each modality, illustrate pre-processing blocks. For example:
- Text: Data retrieval → Language pre-processing (tokenization, cleaning, etc.) → Padding → Word2Vec embedding.
- Audio: Raw audio capture → Signal discretization → Log-mel-spectrogram extraction → Rolling window slicing.
- Video: Video capture → Face detection (with HOG and landmarks) → Face cropping/resizing.
- Model Inference Blocks: Each modality’s processed data is fed into its corresponding deep learning model:
- Text Model: CNN + LSTM architecture.
- Audio Model: Time Distributed CNN → LSTM → Softmax output.
- Video Model: XCeption architecture.
- Integration Layer (WebApp): A central Flask-based web service that accepts processed inputs (or invokes prediction functions from separate modules) and then integrates and displays the results.

##### Architectural Patterns and Design Principles
- Modularity & Separation of Concerns: Each modality (text/audio/video) is developed in its own folder with dedicated pipelines. This reinforces the microservices or modular component-based architecture.
- Reusability: The use of shared model files and reproducible preprocessing scripts suggests high reusability.
- Deep Learning Model Deployment: The project uses a mix of CNNs, LSTMs, and in one case a hybrid CNN-LSTM stack, showing attention to the specific sequence and spatial qualities of different input types.
- Integration via WebApp: The use of Flask and static/dynamic templates shows the clear separation between the frontend interface and backend data processing.

##### Incorporate Relevant Technologies, Frameworks, and Libraries
- Python as the primary language.
- Deep Learning frameworks like TensorFlow/Keras (implied by the usage of .h5 model files, LSTM layers, etc.).
- Preprocessing libraries for NLP (such as NLTK or SpaCy) and for audio signal processing.
- OpenCV and dlib or similar libraries for face detection and landmarks in the video pipeline.
- Flask for the web application, with integrations of JavaScript libraries (D3.js among others) for interactive data visualization.
- Additional tools for model training (Jupyter Notebooks and Colab notebooks).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Acquisition Layer
    subgraph "Input Acquisition"
        T1["Text Input"]:::input
        A1["Audio Input"]:::input
        V1["Video Input"]:::input
    end

    %% Text Processing Pipeline
    subgraph "Text Processing Pipeline"
        T1 --> T2["Text Pre-processing"]:::preprocessing
        T2 --> T3["Word2Vec Embedding"]:::preprocessing
        T3 --> T4["CNN-LSTM (Text Model)"]:::model
    end

    %% Audio Processing Pipeline
    subgraph "Audio Processing Pipeline"
        A1 --> A2["Audio Pre-processing"]:::preprocessing
        A2 --> A3["Log-mel-spectrogram Extraction"]:::preprocessing
        A3 --> A4["TimeDistributed_CNN-LSTM (Audio Model)"]:::model
        A3 --> A5["SVM (Audio Model)"]:::model
        A4 --- A6["Audio Model Output"]:::model
        A5 --- A6
    end

    %% Video Processing Pipeline
    subgraph "Video Processing Pipeline"
        V1 --> V2["Video Pre-processing"]:::preprocessing
        V2 --> V3["FaceDetection_&_Resizing"]:::preprocessing
        V3 --> V4["XCeption (Video Model)"]:::model
    end

    %% Planned Ensemble Model and Integration
    %% Merge outputs of Text, Audio, Video pipelines into an Ensemble model (planned) then to the Web Application.
    T4 --> E1["Ensemble Model (Planned)"]:::model
    A6 --> E1
    V4 --> E1
    E1 --> FL["Flask Web Interface"]:::integration
    FL --> VD["ResultVisualization_(HTML/D3.js)"]:::integration

    %% Data Storage for models
    T4 --- ST["Text Models"]:::storage
    A4 --- SA["Audio Models"]:::storage
    A5 --- SA
    V4 --- SV["Video Models"]:::storage
    FL --- SW["WebApp Models"]:::storage

    %% Click Events
    click T4 "https://github.com/maelfabien/multimodal-emotion-recognition/tree/master/02-Text"
    click A4 "https://github.com/maelfabien/multimodal-emotion-recognition/tree/master/01-Audio"
    click A5 "https://github.com/maelfabien/multimodal-emotion-recognition/tree/master/01-Audio"
    click V4 "https://github.com/maelfabien/multimodal-emotion-recognition/tree/master/03-Video"
    click FL "https://github.com/maelfabien/multimodal-emotion-recognition/tree/master/04-WebApp"

    %% Styles
    classDef input fill:#cce5ff,stroke:#003366,stroke-width:1px;
    classDef preprocessing fill:#d4edda,stroke:#155724,stroke-width:1px;
    classDef model fill:#fff3cd,stroke:#856404,stroke-width:1px;
    classDef integration fill:#f8d7da,stroke:#721c24,stroke-width:1px;
    classDef storage fill:#e2e3e5,stroke:#6c757d,stroke-width:1px;
```
