# lex-au/vocalis architecture

> Architecture diagram of the GitHub repository lex-au/vocalis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lex-au/vocalis
- Repository: https://github.com/lex-au/vocalis

## Overview

##### Project type and purpose
- Vocalis is a full-stack, real-time speech-to-speech AI assistant.
- The frontend is a React + TypeScript single-page app for capturing audio, rendering UI, managing sessions, and displaying visual feedback (orb, stars, transcripts, etc.).
- The backend is a FastAPI server that orchestrates audio streaming over WebSockets, performs voice activity detection (VAD), speech-to-text (Whisper), sends prompts to a local LLM API, invokes a TTS API for speech synthesis, processes image uploads through a vision model, and persists conversation sessions to disk.

##### File structure
- Top-level directories:
- backend/: Python FastAPI app, env config, routes/websocket.py, services/{conversation_storage, llm, transcription (with VAD), tts, vision}
- frontend/: Vite-driven React app with src/components (UI), src/services (audio, WebSocket), src/utils, Tailwind config, package.json
- docs/: static HTML
- setup/run/install scripts for Windows and Unix
- conversations/ (session JSON files)
- Key config/build files: backend/requirements.txt, frontend/package.json, tsconfig.json, Vite config, .env
- Architectural pattern: single backend service exposing a WebSocket endpoint, orchestrating multiple sub-services, and a decoupled frontend communicating via WebSocket.

##### README for additional insights
- Confirms low-latency, streaming architecture with barge-in and silence detection.
- Lists external dependencies: local LM Studio OpenAI‐compatible API, Orpheus (or alternative) TTS API, SmolVLM for vision.

##### Detail without overcomplicating
- Every distinct module or service as its own block.
- How messages traverse the system from capture to playback.
- Don’t omit small but essential parts like the config loader or JSON file store.
- Maintain a balance: include module-level granularity but avoid overloading with implementation minutiae.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend"
        direction TB
        AudioService["Audio Capture/Playback Service"]:::frontend
        WSClient["WebSocket Client"]:::frontend
        AssistantOrb["AssistantOrb"]:::frontend
        BackgroundStars["BackgroundStars"]:::frontend
        ChatInterface["ChatInterface"]:::frontend
        PreferencesModal["PreferencesModal"]:::frontend
        Sidebar["Sidebar"]:::frontend
        SessionManager["SessionManager"]:::frontend
        StateHooks["State Management Hooks"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend"
        direction TB
        MainApp["Main Application Entry"]:::backend
        ConfigLoader["Configuration Loader"]:::backend
        WebSocketServer["WebSocket Server / Handler"]:::backend
        TranscriptionService["VAD + Whisper Transcription Service"]:::backend
        LLMClient["LLM Client Service"]:::backend
        TTSClient["TTS Client Service"]:::backend
        VisionService["Vision Service (SmolVLM)"]:::backend
        ConversationStorage["Conversation Storage Service"]:::backend
        PromptTemplates["Prompt Templates & Settings"]:::backend
        RequirementsSpec["Requirements Specification"]:::backend
    end

    %% External Services
    subgraph "External Services"
        direction TB
        ExternalLLM["Local LLM API (LM Studio)"]:::external
        ExternalTTS["TTS API (Orpheus/Kokoro)"]:::external
    end

    %% Storage
    subgraph "Persistent Storage"
        direction TB
        FileSystem["Disk-based JSON Session Files"]:::storage
    end

    %% Connections
    AudioService -->|"audio chunks"| WSClient
    WSClient -->|"WebSocket"| WebSocketServer
    WebSocketServer --> ConfigLoader
    ConfigLoader --> MainApp
    WebSocketServer --> TranscriptionService
    TranscriptionService --> LLMClient
    WebSocketServer --> VisionService
    VisionService --> LLMClient
    LLMClient --> ExternalLLM
    ExternalLLM --> LLMClient
    LLMClient --> TTSClient
    TTSClient --> ExternalTTS
    ExternalTTS --> TTSClient
    TTSClient --> WebSocketServer
    WebSocketServer --> WSClient
    WSClient -->|"audio playback"| AudioService
    WebSocketServer --> ConversationStorage
    ConversationStorage --> FileSystem
    MainApp --> PromptTemplates
    MainApp --> RequirementsSpec

    %% Click Events
    click AudioService "https://github.com/lex-au/vocalis/blob/main/frontend/src/services/audio.ts"
    click WSClient "https://github.com/lex-au/vocalis/blob/main/frontend/src/services/websocket.ts"
    click AssistantOrb "https://github.com/lex-au/vocalis/blob/main/frontend/src/components/AssistantOrb.tsx"
    click BackgroundStars "https://github.com/lex-au/vocalis/blob/main/frontend/src/components/BackgroundStars.tsx"
    click ChatInterface "https://github.com/lex-au/vocalis/blob/main/frontend/src/components/ChatInterface.tsx"
    click PreferencesModal "https://github.com/lex-au/vocalis/blob/main/frontend/src/components/PreferencesModal.tsx"
    click Sidebar "https://github.com/lex-au/vocalis/blob/main/frontend/src/components/Sidebar.tsx"
    click SessionManager "https://github.com/lex-au/vocalis/blob/main/frontend/src/components/SessionManager.tsx"
    click StateHooks "https://github.com/lex-au/vocalis/blob/main/frontend/src/utils/hooks.ts"
    click WebSocketServer "https://github.com/lex-au/vocalis/blob/main/backend/routes/websocket.py"
    click ConfigLoader "https://github.com/lex-au/vocalis/blob/main/backend/config.py"
    click MainApp "https://github.com/lex-au/vocalis/blob/main/backend/main.py"
    click ConversationStorage "https://github.com/lex-au/vocalis/blob/main/backend/services/conversation_storage.py"
    click LLMClient "https://github.com/lex-au/vocalis/blob/main/backend/services/llm.py"
    click TranscriptionService "https://github.com/lex-au/vocalis/blob/main/backend/services/transcription.py"
    click TTSClient "https://github.com/lex-au/vocalis/blob/main/backend/services/tts.py"
    click VisionService "https://github.com/lex-au/vocalis/blob/main/backend/services/vision.py"
    click PromptTemplates "https://github.com/lex-au/vocalis/tree/main/backend/prompts"
    click RequirementsSpec "https://github.com/lex-au/vocalis/blob/main/backend/requirements.txt"

    %% Styles
    classDef frontend fill:#E0F7FA,stroke:#00796B,stroke-width:1px
    classDef backend fill:#E8F5E9,stroke:#388E3C,stroke-width:1px
    classDef external fill:#FFF3E0,stroke:#F57C00,stroke-width:1px
    classDef storage fill:#F3E5F5,stroke:#8E24AA,stroke-width:1px
```
