How sujalgupta52/realtime-ai-assistant works
Main Components
A. Client-Side (Frontend):
• Web Browser Interface (server/templates/index.html)
• Client-side JavaScript (server/static/js/index.js)
• CSS Styling (server/static/css/style.css)
• Browser's SpeechUtterance API (alternative TTS)
B. Server-Side (Backend):
• Flask Server (server/app.py)
• Core Processing Components:
• Speech-to-Text Controller (server/utils/stt_controller.py)
• LLM Controller (server/utils/llm_controller.py)
• Text-to-Speech Controller (server/utils/tts_controller.py)
C. External Models/Services:
• Whisper Model (STT)
• Llama 3 8B Model (NLP)
• PiperTTS Model (TTS)
[Client Layer]
• ↕
[Server Layer]
• ↕
[Model Layer]
Component Relationships
• User Audio Input → Browser → Flask Server → STT Controller → Whisper Model
• Transcribed Text → LLM Controller → Llama 3 Model → Response Text
• Response Text → TTS Controller → PiperTTS/Browser TTS → Audio Output
Specific Implementation Details
Technical specifications:
• RTX 3060 Hardware Requirements
• 2-3 Second Latency Indicator
• CUDA/CUDNN Integration
• WebSocket or HTTP Communication Paths
• Model File Locations
Additional Elements
• Configuration Settings
• Static Asset Management
• Utility Functions (utils_func.py)
• Error Handling Flows
[User Interaction Layer]
• ↓ Audio Input
• ↑ Audio Output
[Processing Layer]
• → Speech-to-Text Processing
• → Natural Language Processing
• → Text-to-Speech Synthesis
[Model Integration Layer]
• → Model Loading and Management
• → Hardware Acceleration
Important Notes
• Low-latency processing path
• Potential scaling points
• Future expansion points (for TODO items)
• Error handling paths
• Where context management will be implemented (future feature)
Technical Details to Highlight
• Flask routing structure
• Model initialization and loading
• Audio processing pipeline
• Real-time communication channels
• Browser API integration points
Remember to:
• Use consistent notation
• Performance metrics where relevant
• Both synchronous and asynchronous operations
• Potential bottlenecks and optimization points
This system design should effectively communicate the architecture while highlighting the real-time processing capabilities and AI integration points that make this project unique.