# asiff00/on-device-speech-to-speech-conversational-ai architecture

> Architecture diagram of the GitHub repository asiff00/on-device-speech-to-speech-conversational-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/asiff00/on-device-speech-to-speech-conversational-ai
- Repository: https://github.com/asiff00/on-device-speech-to-speech-conversational-ai

## Overview

##### Project type and purpose
- This is an on-device, real-time speech-to-speech conversational AI system.
- It is not a web app or multi-service microservices suite; rather it is an all-local, multi-threaded pipeline that processes live audio, runs voice-activity detection (VAD), speech recognition (ASR), language modeling (LM), text chunking, and text-to-speech (TTS) synthesis, then outputs audio.

##### File structure
- Top level:
- `speech_to_speech.py` – main entry point that ties everything together.
- `text_to_speech.py` – likely a helper/test script for only the TTS side.
- `data/` – pre-downloaded model weights (VAD, ASR, TTS) and voice profiles.
- `src/`
· `config/` – JSON or environment-based configuration files.
· `core/kokoro.py` – integration layer for the Kokoro TTS model.
· `models/` – definitions for ISTFTNet, PLBERT, and generic model wrappers.
· `utils/` – audio I/O, queue management, VAD, ASR, text chunking, LLM calls, and voice selection.
- `.env.template`, `.gitignore`, `requirements.txt`, `assets/` (demo video, architecture SVG)

##### Extract architecture insights from README
- Multi-threaded, continuous streaming architecture with two queue types: audio and text.
- Pipeline stages (in order): VAD → Whisper ASR → Ollama LM → TextChunker → Kokoro TTS → Audio playback.
- Interrupt mechanism: user can press Ctrl+C to interrupt the current response.
- Priority-based chunking and “filler word” technique to reduce perceived latency.

##### Main components
a. Audio Input (Microphone)
b. Voice Activity Detection (Pyannote)
c. ASR (OpenAI Whisper-tiny.en)
d. Text Queue
f. TextChunker (priority-based splitting)
g. TTS (Kokoro-82M via hexgrad/Kokoro-82M) and fallback to eSpeak NG
h. Audio Queue
i. Audio Output (Speakers)
j. Configuration Layer (.env, config.json)
k. Data Storage (model files in `data/models` and `data/voices`)

##### Relationships and data flow
1) Microphone → VAD
2) VAD → ASR
3) ASR → Text Queue
4) Text Queue → LM Server
5) LM Server → TextChunker
6) TextChunker → TTS Engine Queue
7) TTS Engine → Audio Queue
8) Audio Queue → Speaker

##### Architectural patterns and design principles
- Producer-Consumer Queues: separate threads producing audio/text and consuming them downstream.
- Streaming pipeline: downstream tasks begin as soon as data is available.
- Interruptibility: non-blocking queue flush or signal handling to stop current synthesis.
- Modular design: each stage is its own component (module in `src/utils` or `src/core`).
- Configuration externalized via `.env` and `config.json`.

##### Technologies, frameworks, and libraries
- Pyannote for VAD
- OpenAI Whisper (Whisper-tiny.en) for ASR
- Ollama (qwen2.5) for LM
- Kokoro (hexgrad/Kokoro-82M) for TTS
- Python 3.8+ with threading, queue
- eSpeak NG as a fallback voice synthesizer

##### Final advice
- Be as detailed as possible: break down the pipeline into its smallest logical modules.
- Both the static elements (data files, config) and dynamic flow (streaming, queues, threading).
- Use the provided system_architecture.svg as inspiration for layout and styling.
- Review against the README’s “Loop (VAD → Whisper → LM → TextChunker → TTS)” to ensure full coverage.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry Points
    Entry["speech_to_speech.py\n(Main Orchestrator)"]:::orchestrator
    Helper["text_to_speech.py\n(TTS Only Script)"]:::script

    %% Configuration Layer
    subgraph "Configuration"
        Env[".env.template\n(Environment)"]:::config
        ConfigJSON["src/config/config.json"]:::config
        ConfigLoader["src/utils/config.py"]:::config
    end

    %% Data Storage
    subgraph "Data Storage"
        Models["Model Weights\n(data/models/kokoro.pth)"]:::storage
        Voices["Voice Profiles\n(data/voices/*.pt)"]:::storage
    end

    %% Audio Pipeline
    subgraph "Audio Pipeline"
        MIC["Microphone\nAudio Capture"]:::audio
        VAD["Voice Activity Detection\n(Pyannote)"]:::audio
    end

    %% Text Pipeline
    subgraph "Text Pipeline"
        ASR["Automatic Speech Recognition\n(Whisper-tiny.en)"]:::text
        TextQ["Text Queue & Commands"]:::text
        LM["Language Model Server\n(Ollama qwen2.5)"]:::external
        Chunker["TextChunker\n(Priority-Based Split)"]:::text
    end

    %% TTS Pipeline
    subgraph "TTS Pipeline"
        TTSQ["TTS Engine Queue"]:::tts
        Kokoro["Kokoro TTS\n(hexgrad/Kokoro-82M)"]:::tts
        Fallback["Fallback Synthesizer\n(eSpeak NG)"]:::tts
        AudioQ["Audio Queue"]:::audio
        SPKR["Speakers\nAudio Playback"]:::audio
    end

    %% Helper TTS Script
    Helper --> Kokoro

    %% Orchestration Flow
    Entry --> MIC
    Entry --> ConfigLoader
    Entry --> Models
    Entry --> Voices

    MIC --> VAD
    VAD --> ASR
    ASR --> TextQ
    TextQ --> LM
    LM --> Chunker
    Chunker --> TTSQ
    TTSQ --> Kokoro
    Chunker --> Fallback
    Kokoro --> AudioQ
    Fallback --> AudioQ
    AudioQ --> SPKR

    %% Interrupt Flow
    userInterrupt((Ctrl+C)) -.-> LM
    userInterrupt -.-> TTSQ

    %% Click Events
    click Entry "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/speech_to_speech.py"
    click Helper "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/text_to_speech.py"
    click Env "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/.env.template"
    click ConfigJSON "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/src/config/config.json"
    click ConfigLoader "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/src/utils/config.py"
    click Models "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/data/models/kokoro.pth"
    click Voices "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/tree/main/data/voices"
    click MIC "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/src/utils/audio_io.py"
    click VAD "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/src/utils/audio_utils.py"
    click ASR "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/src/utils/speech.py"
    click TextQ "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/src/utils/commands.py"
    click LM "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/src/utils/llm.py"
    click Chunker "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/src/utils/text_chunker.py"
    click Kokoro "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/src/core/kokoro.py"
    click Fallback "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/src/utils/voice.py"
    click AudioQ "https://github.com/asiff00/on-device-speech-to-speech-conversational-ai/blob/main/src/utils/audio_queue.py"

    %% Styles
    classDef audio fill:#cce5ff,stroke:#004085,color:#004085
    classDef text fill:#d4edda,stroke:#155724,color:#155724
    classDef tts fill:#fff3cd,stroke:#856404,color:#856404
    classDef storage fill:#e2e3e5,stroke:#6c757d,color:#6c757d
    classDef config fill:#f8d7da,stroke:#721c24,color:#721c24
    classDef orchestrator fill:#d1ecf1,stroke:#0c5460,color:#0c5460
    classDef script fill:#f5c6cb,stroke:#721c24,color:#721c24
    classDef external fill:#d6d8d9,stroke:#343a40,color:#343a40
```
