How asiff00/on-device-speech-to-speech-conversational-ai works
Project type and purpose
• This is an on-device, real-time speech-to-speech conversational AI system.
• It is not a web app or multi-service microservices suite; rather it is an all-local, multi-threaded pipeline that processes live audio, runs voice-activity detection (VAD), speech recognition (ASR), language modeling (LM), text chunking, and text-to-speech (TTS) synthesis, then outputs audio.
File structure
• Top level:
• speech_to_speech.py – main entry point that ties everything together.
• text_to_speech.py – likely a helper/test script for only the TTS side.
• data/ – pre-downloaded model weights (VAD, ASR, TTS) and voice profiles.
• src/
· config/ – JSON or environment-based configuration files.
· core/kokoro.py – integration layer for the Kokoro TTS model.
· models/ – definitions for ISTFTNet, PLBERT, and generic model wrappers.
· utils/ – audio I/O, queue management, VAD, ASR, text chunking, LLM calls, and voice selection.
• .env.template, .gitignore, requirements.txt, assets/ (demo video, architecture SVG)
Extract architecture insights from README
• Multi-threaded, continuous streaming architecture with two queue types: audio and text.
• Pipeline stages (in order): VAD → Whisper ASR → Ollama LM → TextChunker → Kokoro TTS → Audio playback.
• Interrupt mechanism: user can press Ctrl+C to interrupt the current response.
• Priority-based chunking and “filler word” technique to reduce perceived latency.
Main components
a. Audio Input (Microphone)
b. Voice Activity Detection (Pyannote)
c. ASR (OpenAI Whisper-tiny.en)
d. Text Queue
f. TextChunker (priority-based splitting)
g. TTS (Kokoro-82M via hexgrad/Kokoro-82M) and fallback to eSpeak NG
h. Audio Queue
i. Audio Output (Speakers)
j. Configuration Layer (.env, config.json)
k. Data Storage (model files in data/models and data/voices)
Relationships and data flow
1) Microphone → VAD
2) VAD → ASR
3) ASR → Text Queue
4) Text Queue → LM Server
5) LM Server → TextChunker
6) TextChunker → TTS Engine Queue
7) TTS Engine → Audio Queue
8) Audio Queue → Speaker
Architectural patterns and design principles
• Producer-Consumer Queues: separate threads producing audio/text and consuming them downstream.
• Streaming pipeline: downstream tasks begin as soon as data is available.
• Interruptibility: non-blocking queue flush or signal handling to stop current synthesis.
• Modular design: each stage is its own component (module in src/utils or src/core).
• Configuration externalized via .env and config.json.
Technologies, frameworks, and libraries
• Pyannote for VAD
• OpenAI Whisper (Whisper-tiny.en) for ASR
• Ollama (qwen2.5) for LM
• Kokoro (hexgrad/Kokoro-82M) for TTS
• Python 3.8+ with threading, queue
• eSpeak NG as a fallback voice synthesizer
Final advice
• Be as detailed as possible: break down the pipeline into its smallest logical modules.
• Both the static elements (data files, config) and dynamic flow (streaming, queues, threading).
• Use the provided system_architecture.svg as inspiration for layout and styling.
• Review against the README’s “Loop (VAD → Whisper → LM → TextChunker → TTS)” to ensure full coverage.