How maelfabien/multimodal-emotion-recognition works
Main Components
• Text Analysis Module – Handles natural language processing (NLP), including custom text pre-processing (tokenization, cleaning, lemmatization, etc.), Word2Vec embedding, and a hybrid CNN-LSTM based neural network for personality trait prediction.
• Audio Analysis Module – Manages speech emotion recognition by processing voice recordings through audio signal discretization, log-mel-spectrogram extraction, and a deep learning architecture (using a Time Distributed CNN followed by LSTM layers) as well as an alternative SVM-based approach.
• Video Analysis Module – Involves video-based emotion detection. It first processes video input through face detection (using HOG features and facial landmark detection), resizes the face images and then applies an XCeption-based CNN model to recognize emotions.
• Web Application (Flask) – Orchestrates the interaction between the user and the backend. It integrates the outputs of text, audio, and video pipelines, displays results through its HTML templates, and may trigger additional data visualization (with D3.js and other JS libraries).
• (Planned) Ensemble Model – Although not implemented in the current release, it is intended to combine decisions from text, audio, and video processing pipelines.
• Input Layer(s): Show separate entry points for text, audio, and video data.
• Pre-processing Pipelines: For each modality, illustrate pre-processing blocks. For example:
• Text: Data retrieval → Language pre-processing (tokenization, cleaning, etc.) → Padding → Word2Vec embedding.
• Audio: Raw audio capture → Signal discretization → Log-mel-spectrogram extraction → Rolling window slicing.
• Video: Video capture → Face detection (with HOG and landmarks) → Face cropping/resizing.
• Model Inference Blocks: Each modality’s processed data is fed into its corresponding deep learning model:
• Text Model: CNN + LSTM architecture.
• Audio Model: Time Distributed CNN → LSTM → Softmax output.
• Video Model: XCeption architecture.
• Integration Layer (WebApp): A central Flask-based web service that accepts processed inputs (or invokes prediction functions from separate modules) and then integrates and displays the results.
Architectural Patterns and Design Principles
• Modularity & Separation of Concerns: Each modality (text/audio/video) is developed in its own folder with dedicated pipelines. This reinforces the microservices or modular component-based architecture.
• Reusability: The use of shared model files and reproducible preprocessing scripts suggests high reusability.
• Deep Learning Model Deployment: The project uses a mix of CNNs, LSTMs, and in one case a hybrid CNN-LSTM stack, showing attention to the specific sequence and spatial qualities of different input types.
• Integration via WebApp: The use of Flask and static/dynamic templates shows the clear separation between the frontend interface and backend data processing.
Incorporate Relevant Technologies, Frameworks, and Libraries
• Python as the primary language.
• Deep Learning frameworks like TensorFlow/Keras (implied by the usage of .h5 model files, LSTM layers, etc.).
• Preprocessing libraries for NLP (such as NLTK or SpaCy) and for audio signal processing.
• OpenCV and dlib or similar libraries for face detection and landmarks in the video pipeline.
• Flask for the web application, with integrations of JavaScript libraries (D3.js among others) for interactive data visualization.
• Additional tools for model training (Jupyter Notebooks and Colab notebooks).