How codingaslu/voice-vision-assistant-for-blind works
Project type and purpose
• This is a Python-based, multi-component “Voice & Vision Assistant” for blind and visually impaired users.
• It’s not a traditional web front-end/backend split, but a single application with distinct pipelines:
• Voice I/O (speech-to-text & text-to-speech)
• Function routing (natural language understanding + tool invocation)
• Vision processing (camera frames → people detection → scene analysis)
• External service connectors (search, places, calendar, contacts, email, avatar)
File structure
• Root
• app.py: CLI entry point (download, start, dev)
• requirements.txt, .env.example: dependencies & config
• src/
• main.py: core “agent” orchestration (router logic)
• config.py: environment and API key loading
• utils.py: common helpers for Google API integration
• tools/: modular service connectors
– visual.py: capture & pass images to models
– groq_handler.py: Llama-4-Scout API wrapper
– internet_search.py, google_places.py, calendar.py, communication.py: each external‐API integration
README architecture section
• User → Voice input → Function Router → QueryType decision
• Branches for Visual, Search, Places, Calendar, Contact, Email, General
• Visual path splits on ModelChoice (GPT-4o vs LLAMA)
• Calls external APIs or models, then consolidates to a TTS response
• Virtual avatar via LiveKit optionally sits at output
Main components
a. User Interface Layer
• Microphone & camera (client hardware)
• app.py frontend CLI / LiveKit agent (optional avatar UI)
b. Voice Pipeline
• Deepgram STT → plain text
• ElevenLabs TTS ← spoken response
c. Language & Routing
• Function Router (src/main.py)
• QueryType decision node
d. Vision Processing
• Camera frame ingest (visual.py)
• People detection (Groq API via groq_handler.py)
• Scene analysis (GPT-4o via OpenAI API or LLAMA analysis)
• Internet search (internet_search.py)
• Google Places (google_places.py)
• Calendar management (calendar.py)
• Contacts & Email (communication.py)
f. External Services
• OpenAI API (GPT-4o)
• Groq API (Llama)
• Google Places, Calendar, Contacts, Gmail APIs
• LiveKit API (avatar channel)
g. Data & Config
• src/config.py reading .env
• credentials.json, token.json (OAuth tokens)
Relationships and data flows
1. User “Voice Input” → Deepgram STT → Router
2. Router → QueryType decision
3. QueryType branches → appropriate tool or vision pipeline
4. Vision pipeline → ModelChoice decision → GPT-4o or LLAMA
5. Each tool or model → unified “Response aggregator”
6. Aggregator → ElevenLabs TTS → User “Voice Output”
7. Optional path: Response aggregator → LiveKit avatar channel
Architectural patterns & design principles
• Pipeline pattern: each step is a discreet module (STT, routing, tool, TTS)
• Function routing: dynamic dispatch based on intent
• Strategy pattern: ModelChoice selects analysis strategy (GPT vs LLAMA)
• Modular connectors: each external service in its own file
• Single process orchestration—no microservices, but clear separation of concerns
Final checklist for the principal software engineer
• Break the app into at least these components: Voice I/O, Router, Vision, Tools, External APIs
• Two levels of branching under Vision (ModelChoice)
• External dependency (OpenAI, Groq, Google APIs, LiveKit, Deepgram, ElevenLabs)