How ajayrajrthd/voice-assistant works
Project Type and Purpose
• This is a Python-based, full-stack Voice Assistant application (per “Voice-Assistant” and the presence of app.py, templates/index.html, speech_processing and tools modules).
• It provides a web UI (Flask or similar) that captures user speech, converts it to text, feeds it to an AI agent, invokes domain-specific “tools” (calendar, contacts, email, web/knowledge search), then converts the AI’s response back to speech.
File Structure Analysis
• Top-level:
• app.py / main.py – entry points (web server and/or CLI launcher)
• requirements.txt, .env – dependencies and configuration
• templates/index.html – web-based UI
• scripts/ – build utilities for your knowledge-base index
• src/ – core application code, subdivided into:
– agents/agent.py – orchestrates prompts and decision-making
– prompts/prompts.py – static or dynamic prompt templates
– speech_processing/ – speech_to_text.py, text_to_speech.py, conversation_manager.py
– tools/ – base_tool.py plus subpackages for calendar, contacts, emails, search
– utils.py – shared helpers
• Patterns & Deployment:
• Modular “tool” plugin architecture for adding new capabilities
• Scripts to build and fetch vector indexes (knowledge base)
• Likely a Flask or FastAPI framework in app.py, using Jinja templates
README Insights
• Minimal, but the name and file tree imply a voice-first assistant
Main Components
a. Frontend (Web UI)
• Browser or embedded client rendering templates/index.html
• Captures audio input; plays back audio output
b. Backend Application
• app.py / main.py hosting HTTP endpoints or socket handlers
• ConversationManager coordinating STT → Agent → Tools → TTS
c. Speech Processing Layer
• speech_to_text.py (calls external STT API or local model)
• text_to_speech.py (TTS engine)
d. Agent Core
• agent.py uses prompts.py and LLM API (OpenAI, local model)
e. Tool Plugins
• calendar_tool.py, add/fetch_contact_tool.py, emailing_tool.py, search_web_tool.py, knowledge_base_tool.py
• Each implements a standardized interface (base_tool.py)
f. Data/Storage & External Services
• Knowledge-base index (built by scripts/create_index.py, scripts/fetch_index.py)
• External APIs: Calendar provider, Email SMTP/IMAP, Contacts datastore, Web search or vector store
g. Optional Components
• utils.py for logging, config
• requirements.txt / .env for deployment config
• venv for isolated environment
Component Interactions (Data Flow)
1. User speaks → Web UI captures audio → sends to Backend
2. Backend → speech_to_text → text transcript
3. Transcript → Agent → decides if a “tool” call is needed
4. If tool call: Agent → Tool plugin → External service or KB → returns structured data
5. Agent assembles final response → conversation_manager
6. conversation_manager → text_to_speech → audio output
7. Audio sent back to UI → played to user
Architectural Patterns & Principles
• Modular plugin architecture for tools (open/closed principle)
• Separation of concerns: UI vs. Speech Processing vs. Agent vs. Tools vs. Data
• Single-responsibility modules (each tool handles one domain)
• Scripts decouple index-creation from runtime