How lex-au/vocalis works
Project type and purpose
• Vocalis is a full-stack, real-time speech-to-speech AI assistant.
• The frontend is a React + TypeScript single-page app for capturing audio, rendering UI, managing sessions, and displaying visual feedback (orb, stars, transcripts, etc.).
• The backend is a FastAPI server that orchestrates audio streaming over WebSockets, performs voice activity detection (VAD), speech-to-text (Whisper), sends prompts to a local LLM API, invokes a TTS API for speech synthesis, processes image uploads through a vision model, and persists conversation sessions to disk.
File structure
• Top-level directories:
• backend/: Python FastAPI app, env config, routes/websocket.py, services/{conversation_storage, llm, transcription (with VAD), tts, vision}
• frontend/: Vite-driven React app with src/components (UI), src/services (audio, WebSocket), src/utils, Tailwind config, package.json
• docs/: static HTML
• setup/run/install scripts for Windows and Unix
• conversations/ (session JSON files)
• Key config/build files: backend/requirements.txt, frontend/package.json, tsconfig.json, Vite config, .env
• Architectural pattern: single backend service exposing a WebSocket endpoint, orchestrating multiple sub-services, and a decoupled frontend communicating via WebSocket.
README for additional insights
• Confirms low-latency, streaming architecture with barge-in and silence detection.
• Lists external dependencies: local LM Studio OpenAI‐compatible API, Orpheus (or alternative) TTS API, SmolVLM for vision.
Detail without overcomplicating
• Every distinct module or service as its own block.
• How messages traverse the system from capture to playback.
• Don’t omit small but essential parts like the config loader or JSON file store.
• Maintain a balance: include module-level granularity but avoid overloading with implementation minutiae.