# koljab/realtimevoicechat architecture

> Architecture diagram of the GitHub repository koljab/realtimevoicechat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/koljab/realtimevoicechat
- Repository: https://github.com/koljab/realtimevoicechat

## Overview

##### Project type and purpose
- This is a full-stack, real-time AI Voice Chat application.
- Purpose: Enable natural, low-latency spoken conversation between a browser client and an LLM backend, streaming audio in both directions.

##### File structure
- Top level: Dockerfile, docker-compose.yml, entrypoint.sh/install scripts, requirements.txt, wheels folder → indicates containerized deployment.
- code/ directory:
– server.py: FastAPI app exposing WebSocket endpoints and static file routing.
– speech_pipeline_manager.py: Orchestrates the pipeline stages (STT → LLM → TTS).
– transcribe.py, turndetect.py, upsample_overlap.py, audio_in.py: Speech-to-text and silence/turn detection modules.
– llm_module.py, text_context.py, text_similarity.py: Interfaces with LLM providers and context management.
– audio_module.py, ttsPlaybackProcessor.js: Text-to-speech engines and playback processing.
– static/: Frontend (index.html, app.js) and Web Audio API worklets (pcmWorkletProcessor.js, ttsPlaybackProcessor.js).
- Configuration: system_prompt.txt, reference_audio.*, logsetup.py.

##### Key insights from README
- Real-time streaming over WebSockets.
- Pipeline: Capture → Stream → Transcribe → LLM → Synthesize → Return → Interrupt handling.
- Backend: Python3 + FastAPI, WebSockets, torch/torchaudio, transformers, RealtimeSTT/TTS.
- Frontend: Vanilla JS + Web Audio API + AudioWorklets.
- Docker & Docker Compose orchestration (app + Ollama).

##### Main components & relationships
a. Frontend (Browser)
– Components: Microphone capture → pcmWorkletProcessor → WebSocket client in app.js → ttsPlaybackProcessor for playback.
– Captures audio chunks and streams them to the backend; plays streamed TTS audio.
b. Backend (Dockerized FastAPI Service)
– WebSocket endpoint: server.py receives audio chunks.
– SpeechPipelineManager: coordinates modules in sequence.
- STT Module (transcribe.py + RealtimeSTT): Emits partial/full transcripts.
- TurnDetector (turndetect.py): Detects silence or speaker turn.
- LLM Module (llm_module.py): Sends text to Ollama or OpenAI, returns text responses.
- TTS Module (audio_module.py + RealtimeTTS): Synthesizes speech.
- UpsampleOverlap (upsample_overlap.py): Smooths audio streaming overlaps.
– Static file server: Serves the UI.
– Logging & Configuration: logsetup.py, system_prompt.txt.
c. External Services
– LLM Providers: Ollama service (Docker-compose) or OpenAI API.
– Container runtime: Docker + NVIDIA toolkit for GPU acceleration.

##### Important architectural patterns & principles
- Streaming pipeline: asynchronous, event-driven processing of audio and text chunks.
- Modular design: Each stage (STT, turn detection, LLM, TTS) is encapsulated in its own module.
- Pluggable backends: Replace LLM or TTS engine by swapping modules/config.
- Low latency focus: AudioWorklets on the client; incremental STT/TTS streaming.
- Containerization: Isolate dependencies, simplify GPU access and model management.

- Annotations for protocols (WebSocket), ports, and major libraries/frameworks.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Client Layer"
        direction TB
        MicCapture["Microphone Capture"]:::frontend
        PCMWorklet["pcmWorkletProcessor.js"]:::frontend
        IndexHTML["index.html"]:::frontend
        WebSocketClient["WebSocket Client (app.js)"]:::frontend
        TTSPlayback["ttsPlaybackProcessor.js"]:::frontend
    end

    subgraph "API/WebSocket Layer"
        direction TB
        WebSocketEndpoint["WebSocket Endpoint\n(server.py)"]:::api
    end

    subgraph "Processing Pipeline"
        direction TB
        Orchestrator["SpeechPipelineManager\n(speech_pipeline_manager.py)"]:::backend
        STTInput["Audio Input Handler\n(audio_in.py)"]:::backend
        STT["Realtime STT\n(transcribe.py)"]:::backend
        Detector["Turn/Silence Detector\n(turndetect.py)"]:::backend
        LLMMod["LLM Module\n(llm_module.py)"]:::backend
        TextCtx["Context Mgmt\n(text_context.py)"]:::backend
        TextSim["Similarity Ranking\n(text_similarity.py)"]:::backend
        TTSMod["TTS Engine\n(audio_module.py)"]:::backend
        Upsample["Overlap Smoothing\n(upsample_overlap.py)"]:::backend
        Logging["Logging Setup\n(logsetup.py)"]:::backend
        SystemPrompt["System Prompt\n(system_prompt.txt)"]:::backend
        RefAudioJSON["Reference Audio JSON\n(reference_audio.json)"]:::backend
        RefAudioWAV["Reference Audio WAV\n(reference_audio.wav)"]:::backend
    end

    subgraph "External Services"
        direction TB
        Ollama["Ollama Service"]:::external
        OpenAI["OpenAI API"]:::external
        DockerGPU["Docker & GPU Runtime"]:::external
    end

    %% Data Flows
    MicCapture --> PCMWorklet --> WebSocketClient --> WebSocketEndpoint
    WebSocketEndpoint --> Orchestrator
    Orchestrator --> STTInput --> STT --> Detector --> LLMMod --> TTSMod --> Upsample --> WebSocketEndpoint
    WebSocketEndpoint --> WebSocketClient --> TTSPlayback
    Detector -->|triggers context| TextCtx
    Detector -->|triggers ranking| TextSim
    LLMMod -->|uses context| TextCtx
    LLMMod -->|uses ranking| TextSim
    Orchestrator --> Logging
    Orchestrator --> SystemPrompt
    Orchestrator --> RefAudioJSON
    Orchestrator --> RefAudioWAV
    LLMMod --> Ollama
    LLMMod --> OpenAI
    DockerGPU --> Orchestrator
    DockerGPU --> Ollama

    %% Click Events
    click PCMWorklet "https://github.com/koljab/realtimevoicechat/blob/main/code/static/pcmWorkletProcessor.js"
    click IndexHTML "https://github.com/koljab/realtimevoicechat/blob/main/code/static/index.html"
    click WebSocketClient "https://github.com/koljab/realtimevoicechat/blob/main/code/static/app.js"
    click TTSPlayback "https://github.com/koljab/realtimevoicechat/blob/main/code/static/ttsPlaybackProcessor.js"
    click WebSocketEndpoint "https://github.com/koljab/realtimevoicechat/blob/main/code/server.py"
    click Orchestrator "https://github.com/koljab/realtimevoicechat/blob/main/code/speech_pipeline_manager.py"
    click STTInput "https://github.com/koljab/realtimevoicechat/blob/main/code/audio_in.py"
    click STT "https://github.com/koljab/realtimevoicechat/blob/main/code/transcribe.py"
    click Detector "https://github.com/koljab/realtimevoicechat/blob/main/code/turndetect.py"
    click LLMMod "https://github.com/koljab/realtimevoicechat/blob/main/code/llm_module.py"
    click TextCtx "https://github.com/koljab/realtimevoicechat/blob/main/code/text_context.py"
    click TextSim "https://github.com/koljab/realtimevoicechat/blob/main/code/text_similarity.py"
    click TTSMod "https://github.com/koljab/realtimevoicechat/blob/main/code/audio_module.py"
    click Upsample "https://github.com/koljab/realtimevoicechat/blob/main/code/upsample_overlap.py"
    click Logging "https://github.com/koljab/realtimevoicechat/blob/main/code/logsetup.py"
    click SystemPrompt "https://github.com/koljab/realtimevoicechat/blob/main/code/system_prompt.txt"
    click RefAudioJSON "https://github.com/koljab/realtimevoicechat/blob/main/code/reference_audio.json"
    click RefAudioWAV "https://github.com/koljab/realtimevoicechat/blob/main/code/reference_audio.wav"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef backend fill:#d4edda,stroke:#333,stroke-width:1px
    classDef api fill:#f8d7da,stroke:#333,stroke-width:1px
    classDef external fill:#ffeeba,stroke:#333,stroke-width:1px
```
