How activated-intelligence/voice-chat works
1. This repository is a fully browser-side voice assistant built with Next.js App Router + React. There is no custom backend in the repo; the runtime boundary is the browser, with all speech, inference, and playback happening locally on the client.
2. The main orchestration point is src/app/page.tsx. It likely owns the session state, coordinates microphone capture, receives transcripts, sends prompts to the LLM, and triggers spoken responses. layout.tsx and globals.css provide the app shell and styling only.
3. The core user-facing UI is split into small components under src/components/ui/: conversation.tsx, message.tsx, live-waveform.tsx, avatar.tsx, and button.tsx. These are presentation boundaries for chat history, voice activity visualization, speaker identity/role, and primary controls.
4. The audio ingestion pipeline is implemented via browser-native audio processing plus shipped worker assets in public/. public/vad-processor.js is an AudioWorklet-style processor for streaming microphone audio into the VAD stage, while public/stt-worker-esm.js runs speech recognition logic off the main thread.
5. The speech-to-text boundary is local and model-driven. The README states Whisper runs via @huggingface/transformers, with the model downloaded on first use and cached locally. This means transcription is an asynchronous browser workload, not a server API call.
6. The voice activity detection boundary is separate from transcription. Silero VAD detects when the user is speaking so the app can segment utterances and avoid sending silence into STT/LLM. This is important architecturally because it gates when audio is promoted into the transcription pipeline.
7. The LLM subsystem is wrapped in src/hooks/use-webllm.ts. The repo uses WebLLM with a Qwen 1.5B model for local generation. The hook is the abstraction boundary between conversation state and the inference engine, making it the natural replacement point if the app is switched to an external API or a local server.
8. The text-to-speech subsystem is wrapped in src/hooks/use-tts.ts and supported by src/lib/tts.ts. The TTS layer handles turning generated text into audio using Supertonic via @huggingface/transformers, with voice assets bundled in public/voices/*.bin.
9. src/lib/splitter.ts is a small but important pipeline utility: it segments LLM output into chunks suitable for streaming speech synthesis. That makes the assistant feel responsive by allowing TTS to begin before the full response is complete.
10. src/lib/utils.ts is supporting infrastructure shared by the UI/components, while components.json, Tailwind/PostCSS config, and tsconfig.json define the frontend toolchain. These are secondary to the runtime architecture but confirm this is a strongly client-oriented TypeScript application.
11. The data flow is: microphone input → VAD → STT → conversation state → WebLLM response generation → text splitting → TTS synthesis → audio playback, while the UI renders waveform/chat updates in parallel. This is the key diagrammatic backbone of the repo.
12. A major architectural characteristic is asset-heavy local inference. The README calls out large model downloads from Hugging Face CDN and local caching via IndexedDB after first load. The repo’s public/voices binaries are prebundled, but the bigger ML models are fetched lazily in-browser and reused offline afterward.
13. The project is intentionally designed with a replaceable LLM boundary. The README explicitly says the voice pipeline is the main value and the LLM can be swapped for Claude, GPT, Gemini, Ollama, LM Studio, or any OpenAI-compatible endpoint. That makes the assistant pipeline modular: speech and synthesis are local, while generation can be local or remote.
14. From an architecture-diagram perspective, the most important boundaries are: browser UI, audio processing workers, model runtime hooks, bundled voice assets, and external model download sources. There is no persistent server tier in this repo; persistence is limited to local browser caches for downloaded models and assets.
Main entry points: page.tsx Voice Orchestrator · use-tts.ts TTS Hook · layout.tsx App Shell · conversation.tsx Conversation
Components 16
Browser App
Next.js client shell and orchestrator
- Voice Orchestrator · Main browser-side session controller and conversation flow.
src/app/page.tsx - App Shell · Root document structure and app wrapper.
src/app/layout.tsx - Global Styles · Application-wide styling and visual defaults.
src/app/globals.css - Conversation · Conversation history and message stream container.
src/components/ui/conversation.tsx - Message · Single chat or assistant message presentation.
src/components/ui/message.tsx - Waveform · Live voice activity visualization.
src/components/ui/live-waveform.tsx - Avatar · Speaker identity and assistant/user visual marker.
src/components/ui/avatar.tsx - Button · Primary interaction control used by the voice UI.
src/components/ui/button.tsx - Shared Utils · Common frontend utility helpers.
src/lib/utils.ts
Audio Pipeline
Microphone capture, VAD, transcription, and playback
- VAD Processor · Browser audio processor that streams mic frames into voice activity detection.
public/vad-processor.js - STT Worker · Off-main-thread speech recognition entrypoint.
public/stt-worker-esm.js
AI Runtime
Local generation and speech synthesis hooks
- TTS Hook · Speech synthesis orchestration and playback boundary.
src/hooks/use-tts.ts - LLM Hook · Local model generation boundary using WebLLM.
src/hooks/use-webllm.ts - Text Splitter · Chunks generated text for streaming speech synthesis.
src/lib/splitter.ts - TTS Core · Supertonic-based speech synthesis helper logic.
src/lib/tts.ts
Bundled Assets
Worker scripts and voice model binaries
- Voice Assets · Bundled speaker voice binaries used by local synthesis.
public/voices
Connections 17
- Voice Orchestrator → Conversation: renders · Main session state feeds the conversation view.
- Voice Orchestrator → Waveform: updates · Voice activity state drives live waveform feedback.
- Voice Orchestrator → Avatar: renders · Speaker role and state are reflected in the avatar UI.
- Voice Orchestrator → Button: controls · Primary user actions flow through the button control.
- App Shell → Voice Orchestrator: hosts · Root layout hosts the application page.
- App Shell → Global Styles: applies · Global styles are attached through the app shell.
- Conversation → Message: contains · Conversation is composed of individual message cards.
- Voice Orchestrator → VAD Processor: captures · Microphone audio is routed into VAD processing.
- VAD Processor → STT Worker: gates · Detected speech segments are promoted into transcription.
- STT Worker → Voice Orchestrator: emits transcript · Transcribed user speech updates the orchestrator state.
- Voice Orchestrator → LLM Hook: requests reply · Transcript and conversation context are sent for generation.
- LLM Hook → Text Splitter: streams text · Generated output is segmented for incremental speaking.
- Text Splitter → TTS Hook: feeds chunks · Text chunks are sent into the TTS pipeline.
- TTS Hook → TTS Core: delegates · The hook uses lower-level synthesis helpers.
- TTS Core → Voice Assets: reads assets · Voice binaries are loaded for local synthesis.
- TTS Hook → Voice Orchestrator: plays audio · Synthesized speech returns to the assistant session.
- Voice Orchestrator → Shared Utils: shares helpers · Common utilities support the UI and orchestration code.