How soumajitpal12/multimodal_emotion_recognition_chatbot works
Project type and purpose
• This is a full-stack Python web application (Flask) for multi-modal emotion recognition.
• It integrates three emotion-analysis pipelines—facial (images), vocal (audio), and textual (chatbot)—and persists user/chat data.
File structure
• Top level:
– app.py: Flask application entry point, defines routes and orchestrates all services
– chatbot.py / emotionChatbot/: text-based emotion/chat service
– emotion_model.py + .h5/.json files: pre-trained audio emotion model
– images/: face-emotion model code (face_emotion.py) and train/test datasets
– database.py + users.db: SQLite (or similar) persistence layer for users and chat history
– static/ + templates/: frontend assets (CSS, JS, HTML)
– chat_history.json: optional log store for conversational data
– trainmodel.ipynb, notebook/: development/training artifacts, not part of runtime
README
• Confirms integration of facial, voice, and text emotion modules.
Main components & their responsibilities
1. Client (Web Browser)
– UI pages (templates/*.html) served by Flask
– Static assets (CSS/JS) for styling, audio playback, Vanta.js background
2. Web Server (Flask app.py)
– Routes:
• /facial_emotion → invokes images/face_emotion.py
• /voice_emotion → invokes emotion_model.py
• /text_chatbot → invokes chatbot.py / emotionChatbot module
• /register, /login, /contact_thanks, /home, etc.
– Orchestrator: parses user input (image file, recorded audio blob, chat text), invokes the right module, aggregates results
3. Facial Emotion Module
– face_emotion.py loads a Keras/TensorFlow model (emotiondetector.h5 + .json)
– Uses OpenCV or PIL to preprocess uploaded/streamed image, classifies to one of six emotions
4. Voice Emotion Module
– emotion_model.py loads a separate .h5/.json model for speech-based emotion inference
– Uses librosa or similar for feature extraction on audio clips
5. Textual Emotion/Chatbot Module
– chatbot.py and emotionChatbot folder: NLP engine that maps user text to an empathetic response, may use rule-based or trained embeddings
– Maintains conversation context in chat_history.json or users.db
6. Database Layer (database.py + users.db)
– ORM or raw SQL to store user profiles, credentials, chat transcripts
7. External/Supporting Services
– File system (for model files, image datasets)
– Jupyter training notebooks (offline, not part of serving)
Relationships & data flow
• Client ↔ Flask Server (HTTP requests/responses)
• Flask Server → ML Modules (in-process function calls)
• ML Modules → return emotion results to Flask Server
• Flask Server → Database Layer (read/write user/chat data)
• Flask Server → Client (rendered HTML or JSON, plus optional audio response via static/output.mp3)
Architectural patterns & design principles
• Monolithic but modular—each modality is a module in the same runtime
• MVC-like separation:
– Model: ML models, database.py
– View: Jinja2 templates (templates/*.html), static assets
– Controller: Flask routes in app.py
• Single source of truth for user data (SQLite)
• Clear separation of concerns: preprocessing, inference, persistence, presentation
Essential elements
• Component names and roles
• Clear boundaries (e.g., “Web Client”, “Server Process”, “Database”)
• Notes on technologies: Flask (Python), Keras/TensorFlow, OpenCV/librosa, SQLite