# sujalgupta52/realtime-ai-assistant architecture

> Architecture diagram of the GitHub repository sujalgupta52/realtime-ai-assistant, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sujalgupta52/realtime-ai-assistant
- Repository: https://github.com/sujalgupta52/realtime-ai-assistant

## Overview

##### Main Components

A. Client-Side (Frontend):
- Web Browser Interface (server/templates/index.html)
- Client-side JavaScript (server/static/js/index.js)
- CSS Styling (server/static/css/style.css)
- Browser's SpeechUtterance API (alternative TTS)

B. Server-Side (Backend):
- Flask Server (server/app.py)
- Core Processing Components:
- Speech-to-Text Controller (server/utils/stt_controller.py)
- LLM Controller (server/utils/llm_controller.py)
- Text-to-Speech Controller (server/utils/tts_controller.py)

C. External Models/Services:
- Whisper Model (STT)
- Llama 3 8B Model (NLP)
- PiperTTS Model (TTS)

**[Client Layer]**
- ↕
**[Server Layer]**
- ↕
**[Model Layer]**

##### Component Relationships

- User Audio Input → Browser → Flask Server → STT Controller → Whisper Model
- Transcribed Text → LLM Controller → Llama 3 Model → Response Text
- Response Text → TTS Controller → PiperTTS/Browser TTS → Audio Output

##### Specific Implementation Details

Technical specifications:
- RTX 3060 Hardware Requirements
- 2-3 Second Latency Indicator
- CUDA/CUDNN Integration
- WebSocket or HTTP Communication Paths
- Model File Locations

##### Additional Elements

- Configuration Settings
- Static Asset Management
- Utility Functions (utils_func.py)
- Error Handling Flows

**[User Interaction Layer]**
- ↓ Audio Input
- ↑ Audio Output

**[Processing Layer]**
- → Speech-to-Text Processing
- → Natural Language Processing
- → Text-to-Speech Synthesis

**[Model Integration Layer]**
- → Model Loading and Management
- → Hardware Acceleration

##### Important Notes

- Low-latency processing path
- Potential scaling points
- Future expansion points (for TODO items)
- Error handling paths
- Where context management will be implemented (future feature)

##### Technical Details to Highlight

- Flask routing structure
- Model initialization and loading
- Audio processing pipeline
- Real-time communication channels
- Browser API integration points

Remember to:
- Use consistent notation
- Performance metrics where relevant
- Both synchronous and asynchronous operations
- Potential bottlenecks and optimization points

This system design should effectively communicate the architecture while highlighting the real-time processing capabilities and AI integration points that make this project unique.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles and Classes
    classDef frontend fill:#3498db,stroke:#2980b9,color:white
    classDef backend fill:#2ecc71,stroke:#27ae60,color:white
    classDef model fill:#9b59b6,stroke:#8e44ad,color:white
    classDef api fill:#e67e22,stroke:#d35400,color:white
    classDef config fill:#95a5a6,stroke:#7f8c8d,color:white

    %% Client Layer
    subgraph Client
        UI["Web Interface"]:::frontend
        JS["Client JavaScript"]:::frontend
        CSS["Styling"]:::frontend
        BrowserTTS["Browser TTS API"]:::api
    end

    %% Server Layer
    subgraph Server
        Flask["Flask Server"]:::backend
        
        subgraph Controllers
            STT["Speech-to-Text Controller"]:::backend
            LLM["LLM Controller"]:::backend
            TTS["Text-to-Speech Controller"]:::backend
            Utils["Utility Functions"]:::backend
            Main["Main Processing"]:::backend
        end
        
        subgraph Config
            Settings["Configuration Settings"]:::config
            ErrorHandler["Error Handling"]:::config
        end
    end

    %% Model Layer
    subgraph AI_Models
        WhisperModel["Whisper Model"]:::model
        LlamaModel["Llama 3 8B Model"]:::model
        PiperModel["PiperTTS Model"]:::model
    end

    %% Relationships
    UI --> JS
    UI --> CSS
    JS --> Flask
    Flask --> Main
    Main --> STT
    Main --> LLM
    Main --> TTS
    STT --> WhisperModel
    LLM --> LlamaModel
    TTS --> PiperModel
    TTS --> BrowserTTS
    BrowserTTS --> UI
    Utils --> Controllers

    %% Click Events
    click UI "https://github.com/SujalGupta52/Realtime-AI-Assistant/blob/main/server/templates/index.html"
    click JS "https://github.com/SujalGupta52/Realtime-AI-Assistant/blob/main/server/static/js/index.js"
    click CSS "https://github.com/SujalGupta52/Realtime-AI-Assistant/blob/main/server/static/css/style.css"
    click Flask "https://github.com/SujalGupta52/Realtime-AI-Assistant/blob/main/server/app.py"
    click STT "https://github.com/SujalGupta52/Realtime-AI-Assistant/blob/main/server/utils/stt_controller.py"
    click LLM "https://github.com/SujalGupta52/Realtime-AI-Assistant/blob/main/server/utils/llm_controller.py"
    click TTS "https://github.com/SujalGupta52/Realtime-AI-Assistant/blob/main/server/utils/tts_controller.py"
    click Utils "https://github.com/SujalGupta52/Realtime-AI-Assistant/blob/main/server/utils/utils_func.py"
    click Main "https://github.com/SujalGupta52/Realtime-AI-Assistant/blob/main/server/utils/main.py"

    %% Legend
    subgraph Legend
        L1["Frontend Components"]:::frontend
        L2["Backend Services"]:::backend
        L3["AI Models"]:::model
        L4["External APIs"]:::api
        L5["Configuration"]:::config
    end
```
