How koljab/realtimevoicechat works
Project type and purpose
• This is a full-stack, real-time AI Voice Chat application.
• Purpose: Enable natural, low-latency spoken conversation between a browser client and an LLM backend, streaming audio in both directions.
File structure
• Top level: Dockerfile, docker-compose.yml, entrypoint.sh/install scripts, requirements.txt, wheels folder → indicates containerized deployment.
• code/ directory:
– server.py: FastAPI app exposing WebSocket endpoints and static file routing.
– speech_pipeline_manager.py: Orchestrates the pipeline stages (STT → LLM → TTS).
– transcribe.py, turndetect.py, upsample_overlap.py, audio_in.py: Speech-to-text and silence/turn detection modules.
– llm_module.py, text_context.py, text_similarity.py: Interfaces with LLM providers and context management.
– audio_module.py, ttsPlaybackProcessor.js: Text-to-speech engines and playback processing.
– static/: Frontend (index.html, app.js) and Web Audio API worklets (pcmWorkletProcessor.js, ttsPlaybackProcessor.js).
• Configuration: system_prompt.txt, reference_audio.*, logsetup.py.
Key insights from README
• Real-time streaming over WebSockets.
• Pipeline: Capture → Stream → Transcribe → LLM → Synthesize → Return → Interrupt handling.
• Backend: Python3 + FastAPI, WebSockets, torch/torchaudio, transformers, RealtimeSTT/TTS.
• Frontend: Vanilla JS + Web Audio API + AudioWorklets.
• Docker & Docker Compose orchestration (app + Ollama).
Main components & relationships
a. Frontend (Browser)
– Components: Microphone capture → pcmWorkletProcessor → WebSocket client in app.js → ttsPlaybackProcessor for playback.
– Captures audio chunks and streams them to the backend; plays streamed TTS audio.
b. Backend (Dockerized FastAPI Service)
– WebSocket endpoint: server.py receives audio chunks.
– SpeechPipelineManager: coordinates modules in sequence.
• STT Module (transcribe.py + RealtimeSTT): Emits partial/full transcripts.
• TurnDetector (turndetect.py): Detects silence or speaker turn.
• LLM Module (llm_module.py): Sends text to Ollama or OpenAI, returns text responses.
• TTS Module (audio_module.py + RealtimeTTS): Synthesizes speech.
• UpsampleOverlap (upsample_overlap.py): Smooths audio streaming overlaps.
– Static file server: Serves the UI.
– Logging & Configuration: logsetup.py, system_prompt.txt.
c. External Services
– LLM Providers: Ollama service (Docker-compose) or OpenAI API.
– Container runtime: Docker + NVIDIA toolkit for GPU acceleration.
Important architectural patterns & principles
• Streaming pipeline: asynchronous, event-driven processing of audio and text chunks.
• Modular design: Each stage (STT, turn detection, LLM, TTS) is encapsulated in its own module.
• Pluggable backends: Replace LLM or TTS engine by swapping modules/config.
• Low latency focus: AudioWorklets on the client; incremental STT/TTS streaming.
• Containerization: Isolate dependencies, simplify GPU access and model management.
• Annotations for protocols (WebSocket), ports, and major libraries/frameworks.