# codingaslu/voice-vision-assistant-for-blind architecture

> Architecture diagram of the GitHub repository codingaslu/voice-vision-assistant-for-blind, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/codingaslu/voice-vision-assistant-for-blind
- Repository: https://github.com/codingaslu/voice-vision-assistant-for-blind

## Overview

##### Project type and purpose
- This is a Python-based, multi-component “Voice & Vision Assistant” for blind and visually impaired users.
- It’s not a traditional web front-end/backend split, but a single application with distinct pipelines:
- Voice I/O (speech-to-text & text-to-speech)
- Function routing (natural language understanding + tool invocation)
- Vision processing (camera frames → people detection → scene analysis)
- External service connectors (search, places, calendar, contacts, email, avatar)

##### File structure
- Root
- app.py: CLI entry point (download, start, dev)
- requirements.txt, .env.example: dependencies & config
- src/
- main.py: core “agent” orchestration (router logic)
- config.py: environment and API key loading
- utils.py: common helpers for Google API integration
- tools/: modular service connectors
– visual.py: capture & pass images to models
– groq_handler.py: Llama-4-Scout API wrapper
– internet_search.py, google_places.py, calendar.py, communication.py: each external‐API integration

##### README architecture section
- User → Voice input → Function Router → QueryType decision
- Branches for Visual, Search, Places, Calendar, Contact, Email, General
- Visual path splits on ModelChoice (GPT-4o vs LLAMA)
- Calls external APIs or models, then consolidates to a TTS response
- Virtual avatar via LiveKit optionally sits at output

##### Main components
a. User Interface Layer
- Microphone & camera (client hardware)
- app.py frontend CLI / LiveKit agent (optional avatar UI)
b. Voice Pipeline
- Deepgram STT → plain text
- ElevenLabs TTS ← spoken response
c. Language & Routing
- Function Router (src/main.py)
- QueryType decision node
d. Vision Processing
- Camera frame ingest (visual.py)
- People detection (Groq API via groq_handler.py)
- Scene analysis (GPT-4o via OpenAI API or LLAMA analysis)
- Internet search (internet_search.py)
- Google Places (google_places.py)
- Calendar management (calendar.py)
- Contacts & Email (communication.py)
f. External Services
- OpenAI API (GPT-4o)
- Groq API (Llama)
- Google Places, Calendar, Contacts, Gmail APIs
- LiveKit API (avatar channel)
g. Data & Config
- src/config.py reading .env
- credentials.json, token.json (OAuth tokens)

##### Relationships and data flows
1. User “Voice Input” → Deepgram STT → Router
2. Router → QueryType decision
3. QueryType branches → appropriate tool or vision pipeline
4. Vision pipeline → ModelChoice decision → GPT-4o or LLAMA
5. Each tool or model → unified “Response aggregator”
6. Aggregator → ElevenLabs TTS → User “Voice Output”
7. Optional path: Response aggregator → LiveKit avatar channel

##### Architectural patterns & design principles
- Pipeline pattern: each step is a discreet module (STT, routing, tool, TTS)
- Function routing: dynamic dispatch based on intent
- Strategy pattern: ModelChoice selects analysis strategy (GPT vs LLAMA)
- Modular connectors: each external service in its own file
- Single process orchestration—no microservices, but clear separation of concerns

##### Final checklist for the principal software engineer
- Break the app into at least these components: Voice I/O, Router, Vision, Tools, External APIs
- Two levels of branching under Vision (ModelChoice)
- External dependency (OpenAI, Groq, Google APIs, LiveKit, Deepgram, ElevenLabs)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interface Layer
    subgraph "User Interface Layer"
        style UI fill:#D0E8FF,stroke:#0366D6,stroke-width:1px
        UI_HW["Microphone & Camera"]:::interface
        CLI["app.py"]:::interface
    end

    %% Voice Pipeline
    subgraph "Voice Pipeline"
        style VP fill:#E2F7D8,stroke:#228B22,stroke-width:1px
        STT["Deepgram STT"]:::processing
        TTS["ElevenLabs TTS"]:::processing
    end

    %% Language & Routing
    subgraph "Language & Routing"
        style LR fill:#E2F7D8,stroke:#228B22,stroke-width:1px
        Router["Function Router\n(src/main.py)"]:::processing
        QueryType{"QueryType\nDecision"}:::decision
    end

    %% Vision Processing
    subgraph "Vision Processing"
        style VP2 fill:#E2F7D8,stroke:#228B22,stroke-width:1px
        FrameCap["Frame Capture\n(src/tools/visual.py)"]:::processing
        PeopleDet["People Detection\n(src/tools/groq_handler.py)"]:::processing
        ModelChoice{"ModelChoice\nDecision"}:::decision
        GPT4["GPT-4o\n(OpenAI API)"]:::external
        LLAMA["LLAMA\n(Groq API)"]:::external
    end

    %% Tool Modules
    subgraph "Tool Modules"
        style TM fill:#E2F7D8,stroke:#228B22,stroke-width:1px
        Search["Internet Search\n(src/tools/internet_search.py)"]:::processing
        Places["Google Places\n(src/tools/google_places.py)"]:::processing
        Calendar["Calendar\n(src/tools/calendar.py)"]:::processing
        Comm["Contacts & Email\n(src/tools/communication.py)"]:::processing
    end

    %% Response Aggregator
    Aggregator["Response Aggregator"]:::processing

    %% External Services
    subgraph "External Services"
        style ES fill:none,stroke:#FF6347,stroke-dasharray: 5 5,stroke-width:1px
        OpenAI["OpenAI API"]:::external
        GroqAPI["Groq API"]:::external
        GoogleAPIs["Google APIs\n(Places, Calendar,\nContacts, Gmail)"]:::external
        LiveKit["LiveKit API"]:::external
    end

    %% Data & Config
    subgraph "Data & Config"
        style DC fill:#F0F0F0,stroke:#555555,stroke-width:1px
        Config["src/config.py & .env.example"]:::data
        Utils["src/utils.py"]:::data
        Req["requirements.txt"]:::data
        Tokens["credentials.json / token.json"]:::data
    end

    %% Connections
    UI_HW -->|voice_input| STT
    STT -->|text| Router
    CLI --> Router
    Router --> QueryType
    QueryType -->|vision_intent| FrameCap
    FrameCap --> PeopleDet
    PeopleDet --> ModelChoice
    ModelChoice -->|use GPT| GPT4
    ModelChoice -->|use LLAMA| LLAMA
    QueryType -->|search_intent| Search
    QueryType -->|places_intent| Places
    QueryType -->|calendar_intent| Calendar
    QueryType -->|comm_intent| Comm
    GPT4 --> Aggregator
    LLAMA --> Aggregator
    Search --> Aggregator
    Places --> Aggregator
    Calendar --> Aggregator
    Comm --> Aggregator
    Aggregator --> TTS
    TTS -->|voice_output| UI_HW
    Aggregator -->|avatar_stream| LiveKit

    %% Data & Config flows
    Config --> Router
    Utils --> Router
    Req --> CLI
    Tokens --> Config

    %% Click Events
    click CLI "https://github.com/codingaslu/voice-vision-assistant-for-blind/blob/main/app.py"
    click Router "https://github.com/codingaslu/voice-vision-assistant-for-blind/blob/main/src/main.py"
    click FrameCap "https://github.com/codingaslu/voice-vision-assistant-for-blind/blob/main/src/tools/visual.py"
    click PeopleDet "https://github.com/codingaslu/voice-vision-assistant-for-blind/blob/main/src/tools/groq_handler.py"
    click Search "https://github.com/codingaslu/voice-vision-assistant-for-blind/blob/main/src/tools/internet_search.py"
    click Places "https://github.com/codingaslu/voice-vision-assistant-for-blind/blob/main/src/tools/google_places.py"
    click Calendar "https://github.com/codingaslu/voice-vision-assistant-for-blind/blob/main/src/tools/calendar.py"
    click Comm "https://github.com/codingaslu/voice-vision-assistant-for-blind/blob/main/src/tools/communication.py"
    click Config "https://github.com/codingaslu/voice-vision-assistant-for-blind/blob/main/src/config.py"
    click Req "https://github.com/codingaslu/voice-vision-assistant-for-blind/blob/main/requirements.txt"
    click Utils "https://github.com/codingaslu/voice-vision-assistant-for-blind/blob/main/src/utils.py"

    %% Styles
    classDef interface fill:#D0E8FF,stroke:#0366D6,stroke-width:1px
    classDef processing fill:#E2F7D8,stroke:#228B22,stroke-width:1px
    classDef decision fill:#FFE4B5,stroke:#FFA500,stroke-width:1px,stroke-dasharray: 2 2
    classDef external fill:#FFD8D8,stroke:#FF6347,stroke-dasharray: 5 5
    classDef data fill:#F0F0F0,stroke:#555,stroke-width:1px,stroke-dasharray: 2 2
```
