# soumajitpal12/multimodal_emotion_recognition_chatbot architecture

> Architecture diagram of the GitHub repository soumajitpal12/multimodal_emotion_recognition_chatbot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/soumajitpal12/multimodal_emotion_recognition_chatbot
- Repository: https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot

## Overview

##### Project type and purpose
- This is a full-stack Python web application (Flask) for multi-modal emotion recognition.
- It integrates three emotion-analysis pipelines—facial (images), vocal (audio), and textual (chatbot)—and persists user/chat data.

##### File structure
- Top level:
– app.py: Flask application entry point, defines routes and orchestrates all services
– chatbot.py / emotionChatbot/: text-based emotion/chat service
– emotion_model.py + .h5/.json files: pre-trained audio emotion model
– images/: face-emotion model code (face_emotion.py) and train/test datasets
– database.py + users.db: SQLite (or similar) persistence layer for users and chat history
– static/ + templates/: frontend assets (CSS, JS, HTML)
– chat_history.json: optional log store for conversational data
– trainmodel.ipynb, notebook/: development/training artifacts, not part of runtime

##### README
- Confirms integration of facial, voice, and text emotion modules.

##### Main components & their responsibilities
1. Client (Web Browser)
– UI pages (templates/*.html) served by Flask
– Static assets (CSS/JS) for styling, audio playback, Vanta.js background
2. Web Server (Flask app.py)
– Routes:
- /facial_emotion → invokes images/face_emotion.py
- /voice_emotion  → invokes emotion_model.py
- /text_chatbot   → invokes chatbot.py / emotionChatbot module
- /register, /login, /contact_thanks, /home, etc.
– Orchestrator: parses user input (image file, recorded audio blob, chat text), invokes the right module, aggregates results
3. Facial Emotion Module
– face_emotion.py loads a Keras/TensorFlow model (emotiondetector.h5 + .json)
– Uses OpenCV or PIL to preprocess uploaded/streamed image, classifies to one of six emotions
4. Voice Emotion Module
– emotion_model.py loads a separate .h5/.json model for speech-based emotion inference
– Uses librosa or similar for feature extraction on audio clips
5. Textual Emotion/Chatbot Module
– chatbot.py and emotionChatbot folder: NLP engine that maps user text to an empathetic response, may use rule-based or trained embeddings
– Maintains conversation context in chat_history.json or users.db
6. Database Layer (database.py + users.db)
– ORM or raw SQL to store user profiles, credentials, chat transcripts
7. External/Supporting Services
– File system (for model files, image datasets)
– Jupyter training notebooks (offline, not part of serving)

##### Relationships & data flow
- Client ↔ Flask Server (HTTP requests/responses)
- Flask Server → ML Modules (in-process function calls)
- ML Modules → return emotion results to Flask Server
- Flask Server → Database Layer (read/write user/chat data)
- Flask Server → Client (rendered HTML or JSON, plus optional audio response via static/output.mp3)

##### Architectural patterns & design principles
- Monolithic but modular—each modality is a module in the same runtime
- MVC-like separation:
– Model: ML models, database.py
– View: Jinja2 templates (templates/*.html), static assets
– Controller: Flask routes in app.py
- Single source of truth for user data (SQLite)
- Clear separation of concerns: preprocessing, inference, persistence, presentation

##### Essential elements
- Component names and roles
- Clear boundaries (e.g., “Web Client”, “Server Process”, “Database”)
- Notes on technologies: Flask (Python), Keras/TensorFlow, OpenCV/librosa, SQLite

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Web Client Layer
    subgraph "Web Client" 
        direction TB
        Templates["Templates Directory"]:::frontend
        Static["Static Assets"]:::frontend
    end

    %% Server Layer
    subgraph "Server Process"
        direction TB
        Flask["Flask App (app.py)"]:::backend
    end

    %% Emotion Analysis Layer
    subgraph "Emotion Analysis Layer"
        direction TB
        Facial["Facial Emotion Module"]:::ml
        Voice["Voice Emotion Module"]:::ml
        Textual["Text/Chatbot Module"]:::ml
    end

    %% Storage Layer
    subgraph "Storage and Models"
        direction TB
        DB["User DB (SQLite)"]:::db
        FileSys["Model Files & Datasets"]:::fs
        Notebooks["Training Notebooks"]:::fs
    end

    %% Connections
    Templates -->|"HTTP Requests"| Flask
    Static -->|"Assets Load"| Flask
    Flask -->|"Image File"| Facial
    Flask -->|"Audio Blob"| Voice
    Flask -->|"Text Query"| Textual
    Facial -->|"Emotion Result"| Flask
    Voice -->|"Emotion Result"| Flask
    Textual -->|"Response"| Flask
    Flask -->|"User/Chat Data"| DB
    Flask -->|"Loads Models"| FileSys

    %% Click Events
    click Templates "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/tree/main/templates/"
    click Static "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/tree/main/static/"
    click Flask "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/blob/main/app.py"
    click Facial "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/blob/main/images/face_emotion.py"
    click Voice "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/blob/main/emotion_model.py"
    click Textual "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/blob/main/chatbot.py"
    click DB "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/blob/main/database.py"
    click FileSys "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/tree/main/images/"
    click FileSys "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/blob/main/emotiondetector.h5"
    click FileSys "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/blob/main/emotiondetector.json"
    click FileSys "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/blob/main/emotiondetect.h5"
    click FileSys "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/blob/main/emotiondetect.json"
    click Notebooks "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/blob/main/trainmodel.ipynb"
    click Notebooks "https://github.com/soumajitpal12/multimodal_emotion_recognition_chatbot/tree/main/notebook/"

    %% Styles
    classDef frontend fill:#D6E9FE,stroke:#1E90FF,color:#000
    classDef backend fill:#DFF2BF,stroke:#4F8A10,color:#000
    classDef ml fill:#FFE4B5,stroke:#FF8C00,color:#000
    classDef db fill:#E6E6FA,stroke:#8A2BE2,color:#000
    classDef fs fill:#F5F5F5,stroke:#A9A9A9,color:#000
```
