# gkintu/hf-speech-to-speech architecture

> Architecture diagram of the GitHub repository gkintu/hf-speech-to-speech, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gkintu/hf-speech-to-speech
- Repository: https://github.com/gkintu/hf-speech-to-speech

## Overview

This repository is a modular voice-agent stack: browser microphone audio reaches a Realtime-compatible API, passes through speech detection, turn detection, transcription, language generation, and speech synthesis, then returns to the browser. The README documents queue-based stage separation and selectable backends. The graph includes the browser demo, API/session layer, core model stages, and tool calling. Some implementation excerpts are partial, so several arrows reflect the documented workflow rather than inspected call sites.

Main entry points: [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (Pipeline setup), [`src/speech_to_speech/pipeline/messages.py`](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py) (Pipeline messages), [`src/speech_to_speech/LLM/language_model.py`](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py) (Language model), [`demo/main.js`](https://github.com/gkintu/hf-speech-to-speech/blob/main/demo/main.js) (Demo app)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Browser client"]
  node_demo["Demo app<br/>[main.js]"]
  node_wsclient["WebSocket client<br/>[s2s-ws-client.js]"]
  node_rtcclient["WebRTC client<br/>[s2s-rtc-client.js]"]
end

subgraph group_realtime["Realtime API"]
  node_websocket["WebSocket routes"]
  node_rtc["WebRTC sessions<br/>[webrtc_session.py]"]
  node_server["Realtime server<br/>[server.py]"]
  node_service["Session service<br/>[service.py]"]
  node_unit["Pipeline unit<br/>[pipeline_unit.py]"]
end

subgraph group_pipeline["Speech pipeline"]
  node_orchestrator["Pipeline setup<br/>[s2s_pipeline.py]"]
  node_queues["Pipeline messages<br/>[messages.py]"]
end

subgraph group_models["Model backends"]
  node_vad["Speech detection<br/>[vad_handler.py]"]
  node_turn["Turn detection<br/>[smart_turn.py]"]
  node_stt["Transcription"]
  node_llm["Language model<br/>[language_model.py]"]
  node_tools["Tool calling<br/>[function_call.py]"]
  node_tts["Speech synthesis"]
end

node_speaker(("Voice user"))
node_llama["llama.cpp server"]

node_speaker -->|"speaks into"| node_demo
node_demo -->|"selects transport"| node_wsclient
node_demo -->|"selects transport"| node_rtcclient
node_wsclient -->|"streams audio"| node_websocket
node_rtcclient -->|"sends media"| node_rtc
node_websocket -->|"claims session"| node_server
node_rtc -->|"claims session"| node_server
node_server -->|"manages units"| node_unit
node_unit -->|"holds session"| node_service
node_orchestrator -->|"configures service"| node_service
node_orchestrator -->|"configures handler"| node_vad
node_orchestrator -->|"selects backend"| node_stt
node_orchestrator -->|"selects backend"| node_llm
node_orchestrator -->|"selects backend"| node_tts
node_vad -->|"checks utterance"| node_turn
node_vad -->|"emits speech events"| node_queues
node_stt -->|"emits transcript"| node_queues
node_llm -->|"emits response text"| node_queues
node_llm -->|"dispatches calls"| node_tools
node_llm -.->|"requests generation"| node_llama
node_tts -->|"emits audio"| node_queues
node_service -->|"routes events"| node_queues
node_wsclient -->|"plays audio"| node_speaker
node_rtcclient -->|"plays audio"| node_speaker

click node_demo "https://github.com/gkintu/hf-speech-to-speech/blob/main/demo/main.js"
click node_wsclient "https://github.com/gkintu/hf-speech-to-speech/blob/main/demo/ws/s2s-ws-client.js"
click node_rtcclient "https://github.com/gkintu/hf-speech-to-speech/blob/main/demo/rtc/s2s-rtc-client.js"
click node_websocket "https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py"
click node_rtc "https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/webrtc_session.py"
click node_server "https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/server.py"
click node_service "https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/service.py"
click node_unit "https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/pipeline_unit.py"
click node_orchestrator "https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py"
click node_queues "https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py"
click node_vad "https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py"
click node_turn "https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/VAD/smart_turn.py"
click node_stt "https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/STT/parakeet_tdt_handler.py"
click node_llm "https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py"
click node_tools "https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_call.py"
click node_tts "https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_demo,node_wsclient,node_rtcclient,node_speaker toneBlue
class node_websocket,node_rtc,node_server,node_service,node_unit toneAmber
class node_orchestrator,node_queues,node_llama toneMint
class node_vad,node_turn,node_stt,node_llm,node_tools,node_tts toneRose
```

## Components

### Browser client

- [Demo app](https://github.com/gkintu/hf-speech-to-speech/blob/main/demo/main.js): `demo/main.js`
- [WebSocket client](https://github.com/gkintu/hf-speech-to-speech/blob/main/demo/ws/s2s-ws-client.js): `demo/ws/s2s-ws-client.js`
- [WebRTC client](https://github.com/gkintu/hf-speech-to-speech/blob/main/demo/rtc/s2s-rtc-client.js): `demo/rtc/s2s-rtc-client.js`

### Realtime API

- [WebSocket routes](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py): `src/speech_to_speech/api/openai_realtime/websocket_router.py`
- [WebRTC sessions](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/webrtc_session.py): `src/speech_to_speech/api/openai_realtime/webrtc_session.py`
- [Realtime server](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/server.py): `src/speech_to_speech/api/openai_realtime/server.py`
- [Session service](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/service.py): `src/speech_to_speech/api/openai_realtime/service.py`
- [Pipeline unit](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/pipeline_unit.py): `src/speech_to_speech/api/openai_realtime/pipeline_unit.py`

### Speech pipeline

- [Pipeline setup](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py): `src/speech_to_speech/s2s_pipeline.py`
- [Pipeline messages](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py): `src/speech_to_speech/pipeline/messages.py`

### Model backends

- [Speech detection](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py): `src/speech_to_speech/VAD/vad_handler.py`
- [Turn detection](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/VAD/smart_turn.py): `src/speech_to_speech/VAD/smart_turn.py`
- [Transcription](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/STT/parakeet_tdt_handler.py): `src/speech_to_speech/STT/parakeet_tdt_handler.py`
- [Language model](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py): `src/speech_to_speech/LLM/language_model.py`
- [Tool calling](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_call.py): `src/speech_to_speech/LLM/tool_call/function_call.py`
- [Speech synthesis](https://github.com/gkintu/hf-speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py): `src/speech_to_speech/TTS/qwen3_tts_handler.py`

### Other components

- **Voice user**
- **llama.cpp server**

## Connections

- Voice user → Demo app: speaks into
- Demo app → WebSocket client: selects transport
- Demo app → WebRTC client: selects transport
- WebSocket client → WebSocket routes: streams audio
- WebRTC client → WebRTC sessions: sends media
- WebSocket routes → Realtime server: claims session
- WebRTC sessions → Realtime server: claims session
- Realtime server → Pipeline unit: manages units
- Pipeline unit → Session service: holds session
- Pipeline setup → Session service: configures service
- Pipeline setup → Speech detection: configures handler
- Pipeline setup → Transcription: selects backend
- Pipeline setup → Language model: selects backend
- Pipeline setup → Speech synthesis: selects backend
- Speech detection → Turn detection: checks utterance
- Speech detection → Pipeline messages: emits speech events
- Transcription → Pipeline messages: emits transcript
- Language model → Pipeline messages: emits response text
- Language model → Tool calling: dispatches calls
- Language model → llama.cpp server: requests generation
- Speech synthesis → Pipeline messages: emits audio
- Session service → Pipeline messages: routes events
- WebSocket client → Voice user: plays audio
- WebRTC client → Voice user: plays audio