# swayam42/ai-voice-agent architecture

> Architecture diagram of the GitHub repository swayam42/ai-voice-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/swayam42/ai-voice-agent
- Repository: https://github.com/swayam42/ai-voice-agent

## Overview

##### Project Type & Purpose
- This is a full-stack, voice-first conversational AI web application (“Chanakya AI Voice Agent”) built with a FastAPI backend and a browser-based frontend.
- Purpose: record user speech, transcribe via AssemblyAI, generate responses via Google Gemini (with function calling for web search & weather), synthesize speech via Murf.ai, and play back audio—all in a one-tap chat interface with streaming support.

##### File Structure
- Top-level:
- main.py (FastAPI entrypoint)
- services/ (domain logic wrappers for STT, TTS, LLM, streaming, web search, weather)
- schemas/ (Pydantic request/response models)
- templates/ + static/ (HTML, CSS, JS, images, sounds) ⇒ the browser UI
- requirements.txt, .env.example, LICENSE, README
- Patterns:
- Layered separation: Routes (main.py) → Service layer → External APIs
- MVC-like: “View” (templates/static), “Controller” (FastAPI routes), “Model” (Pydantic schemas)
- Streaming via WebSocket (/ws) alongside REST endpoints

##### Key Architecture Insights from README
- Multi-stage pipeline: STT → LLM → TTS
- Endpoints:
- REST POST /agent/chat/{session_id}
- WS   /ws  (streaming partial transcripts & chunked TTS)
- REST POST /tts/echo, /generate_audio, /transcribe/file
- Debug GET endpoints for web search & LLM testing
- In-memory per-session history (CHAT_HISTORY dict)
- API-key gating via UI modal (no shared keys publicly)

##### Main Components & Their Interactions
a. Frontend (Browser)
- index.html + script.js + CSS: Records audio (MediaRecorder), uploads via REST, opens WS for streaming, plays back returned audio, renders chat bubbles & sidebar tools.
b. API Gateway (FastAPI)
- Receives HTTP & WS requests, dispatches to service layer, returns JSON or audio chunks.
c. Service Layer
- stt_service: AssemblyAI REST/streaming transcription
- llm_service: Builds prompts, calls Gemini, handles function calling (web_search, weather)
- tts_service: Murf.ai REST TTS
- murf_ws_service: Murf.ai WebSocket streaming TTS
- web_search_service: Tavily wrapper
- weather_service: OpenWeather wrapper
- streaming_transcriber: AssemblyAI streaming helper
d. Session Store
- In-memory CHAT_HISTORY dictionary (keyed by session_id)
e. External Services
- AssemblyAI (Speech-to-Text)
- Google Gemini LLM + function-calling
- Murf.ai TTS (REST + WS)
- Tavily Web Search
- OpenWeather API

##### Architectural Patterns & Design Principles
- Layered architecture (separation of concerns)
- Single-responsibility services for each external integration
- Pydantic schemas for clear API contracts
- WebSocket for real-time streaming (both STT and TTS)
- In-memory session store for quick prototyping (swap with Redis for production)
- API-key gating for public safety

##### Final Advice
- Distinguish streaming vs non-streaming paths
- How function-calling workflows interleave web search / weather calls
- API-key gating & session handling are visible

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Frontend (Browser UI)"
        A1["index.html"]:::frontend
        A2["script.js"]:::frontend
        A3["style.css"]:::frontend
        A4["images/"]:::frontend
        A5["sounds/"]:::frontend
    end

    subgraph "API Gateway (FastAPI)"
        B1["main.py"]:::backend
    end

    subgraph "Service Layer"
        C1["stt_service.py"]:::service
        C2["streaming_transcriber.py"]:::service
        C3["llm_service.py"]:::service
        C4["tts_service.py"]:::service
        C5["murf_ws_service.py"]:::service
        C6["web_search_service.py"]:::service
        C7["weather_service.py"]:::service
    end

    D1["CHAT_HISTORY (In-Memory Store)"]:::store

    subgraph "External APIs"
        E1["AssemblyAI (STT)"]:::external
        E2["Google Gemini (LLM)"]:::external
        E3["Murf.ai (TTS)"]:::external
        E4["Tavily Web Search"]:::external
        E5["OpenWeather API"]:::external
    end

    subgraph "Config & Docs"
        F1[".env.example"]:::config
        F2["requirements.txt"]:::config
        F3["README.md"]:::config
        F4["LICENSE.txt"]:::config
    end

    A2 -->|"HTTP POST /agent/chat & WS /ws"| B1
    B1 -->|"uses"| C1
    C1 -->|"HTTP POST/WS"| E1
    E1 -->|"transcript text"| C1
    C1 -->|"text output"| B1

    B1 -->|"uses"| C3
    C3 -->|"HTTP JSON"| E2
    E2 -->|"LLM reply & function calls"| C3

    C3 -->|"function call: web_search"| C6
    C6 -->|"HTTP GET"| E4
    E4 -->|"search results"| C6
    C6 --> C3

    C3 -->|"function call: weather"| C7
    C7 -->|"HTTP GET"| E5
    E5 -->|"weather data"| C7
    C7 --> C3

    C3 -->|"response text"| C4
    C4 -->|"HTTP POST"| E3
    E3 -->|"audio bytes"| C4

    C3 -->|"response text"| C5
    C5 -->|"WebSocket"| E3
    E3 -->|"audio chunks"| C5

    C4 -->|"audio"| B1
    C5 -->|"audio stream"| B1
    B1 -->|"audio playback"| A2

    B1 -->|"reads/writes"| D1
    D1 -->|"session history"| B1

    click A1 "https://github.com/swayam42/ai-voice-agent/blob/main/app/templates/index.html"
    click A2 "https://github.com/swayam42/ai-voice-agent/blob/main/app/static/JS/script.js"
    click A3 "https://github.com/swayam42/ai-voice-agent/blob/main/app/static/css/style.css"
    click A4 "https://github.com/swayam42/ai-voice-agent/tree/main/app/static/images/"
    click A5 "https://github.com/swayam42/ai-voice-agent/tree/main/app/static/sounds/"
    click B1 "https://github.com/swayam42/ai-voice-agent/blob/main/app/main.py"
    click C1 "https://github.com/swayam42/ai-voice-agent/blob/main/app/services/stt_service.py"
    click C2 "https://github.com/swayam42/ai-voice-agent/blob/main/app/services/streaming_transcriber.py"
    click C3 "https://github.com/swayam42/ai-voice-agent/blob/main/app/services/llm_service.py"
    click C4 "https://github.com/swayam42/ai-voice-agent/blob/main/app/services/tts_service.py"
    click C5 "https://github.com/swayam42/ai-voice-agent/blob/main/app/services/murf_ws_service.py"
    click C6 "https://github.com/swayam42/ai-voice-agent/blob/main/app/services/web_search_service.py"
    click C7 "https://github.com/swayam42/ai-voice-agent/blob/main/app/services/weather_service.py"
    click F1 "https://github.com/swayam42/ai-voice-agent/blob/main/.env.example"
    click F2 "https://github.com/swayam42/ai-voice-agent/blob/main/requirements.txt"
    click F3 "https://github.com/swayam42/ai-voice-agent/blob/main/README.md"
    click F4 "https://github.com/swayam42/ai-voice-agent/blob/main/LICENSE.txt"

    classDef frontend fill:#D0E7FF,stroke:#1E90FF,color:#000
    classDef backend fill:#DFFFE0,stroke:#32CD32,color:#000
    classDef service fill:#FFF5CC,stroke:#FFD700,color:#000
    classDef external fill:#FFE5CC,stroke:#FF8C00,color:#000
    classDef store fill:#F0F0F0,stroke:#A9A9A9,color:#000
    classDef config fill:#F5F5F5,stroke:#C0C0C0,color:#000
```
