How swayam42/ai-voice-agent works
Project Type & Purpose
• This is a full-stack, voice-first conversational AI web application (“Chanakya AI Voice Agent”) built with a FastAPI backend and a browser-based frontend.
• Purpose: record user speech, transcribe via AssemblyAI, generate responses via Google Gemini (with function calling for web search & weather), synthesize speech via Murf.ai, and play back audio—all in a one-tap chat interface with streaming support.
File Structure
• Top-level:
• main.py (FastAPI entrypoint)
• services/ (domain logic wrappers for STT, TTS, LLM, streaming, web search, weather)
• schemas/ (Pydantic request/response models)
• templates/ + static/ (HTML, CSS, JS, images, sounds) ⇒ the browser UI
• requirements.txt, .env.example, LICENSE, README
• Patterns:
• Layered separation: Routes (main.py) → Service layer → External APIs
• MVC-like: “View” (templates/static), “Controller” (FastAPI routes), “Model” (Pydantic schemas)
• Streaming via WebSocket (/ws) alongside REST endpoints
Key Architecture Insights from README
• Multi-stage pipeline: STT → LLM → TTS
• Endpoints:
• REST POST /agent/chat/{session_id}
• WS /ws (streaming partial transcripts & chunked TTS)
• REST POST /tts/echo, /generate_audio, /transcribe/file
• Debug GET endpoints for web search & LLM testing
• In-memory per-session history (CHAT_HISTORY dict)
• API-key gating via UI modal (no shared keys publicly)
Main Components & Their Interactions
a. Frontend (Browser)
• index.html + script.js + CSS: Records audio (MediaRecorder), uploads via REST, opens WS for streaming, plays back returned audio, renders chat bubbles & sidebar tools.
b. API Gateway (FastAPI)
• Receives HTTP & WS requests, dispatches to service layer, returns JSON or audio chunks.
c. Service Layer
• stt_service: AssemblyAI REST/streaming transcription
• llm_service: Builds prompts, calls Gemini, handles function calling (web_search, weather)
• tts_service: Murf.ai REST TTS
• murf_ws_service: Murf.ai WebSocket streaming TTS
• web_search_service: Tavily wrapper
• weather_service: OpenWeather wrapper
• streaming_transcriber: AssemblyAI streaming helper
d. Session Store
• In-memory CHAT_HISTORY dictionary (keyed by session_id)
e. External Services
• AssemblyAI (Speech-to-Text)
• Google Gemini LLM + function-calling
• Murf.ai TTS (REST + WS)
• Tavily Web Search
• OpenWeather API
Architectural Patterns & Design Principles
• Layered architecture (separation of concerns)
• Single-responsibility services for each external integration
• Pydantic schemas for clear API contracts
• WebSocket for real-time streaming (both STT and TTS)
• In-memory session store for quick prototyping (swap with Redis for production)
• API-key gating for public safety
Final Advice
• Distinguish streaming vs non-streaming paths
• How function-calling workflows interleave web search / weather calls
• API-key gating & session handling are visible