How gkintu/hf-speech-to-speech works
This repository is a modular voice-agent stack: browser microphone audio reaches a Realtime-compatible API, passes through speech detection, turn detection, transcription, language generation, and speech synthesis, then returns to the browser. The README documents queue-based stage separation and selectable backends. The graph includes the browser demo, API/session layer, core model stages, and tool calling. Some implementation excerpts are partial, so several arrows reflect the documented workflow rather than inspected call sites.
Main entry points: s2s_pipeline.py Pipeline setup · messages.py Pipeline messages · language_model.py Language model · main.js Demo app
Components 18
Browser client
- Demo app
demo/main.js - WebSocket client
demo/ws/s2s-ws-client.js - WebRTC client
demo/rtc/s2s-rtc-client.js
Realtime API
- WebSocket routes
src/speech_to_speech/api/openai_realtime/websocket_router.py - WebRTC sessions
src/speech_to_speech/api/openai_realtime/webrtc_session.py - Realtime server
src/speech_to_speech/api/openai_realtime/server.py - Session service
src/speech_to_speech/api/openai_realtime/service.py - Pipeline unit
src/speech_to_speech/api/openai_realtime/pipeline_unit.py
Speech pipeline
- Pipeline setup
src/speech_to_speech/s2s_pipeline.py - Pipeline messages
src/speech_to_speech/pipeline/messages.py
Model backends
- Speech detection
src/speech_to_speech/VAD/vad_handler.py - Turn detection
src/speech_to_speech/VAD/smart_turn.py - Transcription
src/speech_to_speech/STT/parakeet_tdt_handler.py - Language model
src/speech_to_speech/LLM/language_model.py - Tool calling
src/speech_to_speech/LLM/tool_call/function_call.py - Speech synthesis
src/speech_to_speech/TTS/qwen3_tts_handler.py
Other components
- Voice user
- llama.cpp server
Connections 24
- Voice user → Demo app: speaks into
- Demo app → WebSocket client: selects transport
- Demo app → WebRTC client: selects transport
- WebSocket client → WebSocket routes: streams audio
- WebRTC client → WebRTC sessions: sends media
- WebSocket routes → Realtime server: claims session
- WebRTC sessions → Realtime server: claims session
- Realtime server → Pipeline unit: manages units
- Pipeline unit → Session service: holds session
- Pipeline setup → Session service: configures service
- Pipeline setup → Speech detection: configures handler
- Pipeline setup → Transcription: selects backend
- Pipeline setup → Language model: selects backend
- Pipeline setup → Speech synthesis: selects backend
- Speech detection → Turn detection: checks utterance
- Speech detection → Pipeline messages: emits speech events
- Transcription → Pipeline messages: emits transcript
- Language model → Pipeline messages: emits response text
- Language model → Tool calling: dispatches calls
- Language model → llama.cpp server: requests generation
- Speech synthesis → Pipeline messages: emits audio
- Session service → Pipeline messages: routes events
- WebSocket client → Voice user: plays audio
- WebRTC client → Voice user: plays audio