How huggingface/speech-to-speech works
Speech To Speech is a modular, low-latency voice-agent runtime. Its principal workflow takes client audio through voice activity detection, transcription, language-model response generation, and speech synthesis, returning realtime events and audio to the client. The repository also exposes OpenAI Realtime-compatible WebSocket and WebRTC sessions, supports interchangeable model backends, and includes internal turn, cancellation, and typed-message coordination. Edges follow README statements and shown source references; unsampled backend and transport details are represented only at the directory or subsystem level.
Main entry points: s2s_pipeline.py Pipeline orchestrator · chat.py Language model · websocket_router.py WebSocket router · base_stt_handler.py Transcription
Components 21
Realtime API
- Realtime server
src/speech_to_speech/api/openai_realtime/server.py - WebSocket router
src/speech_to_speech/api/openai_realtime/websocket_router.py - Realtime service
src/speech_to_speech/api/openai_realtime/service.py - Session pipeline unit
src/speech_to_speech/api/openai_realtime/pipeline_unit.py - LLM proxy
src/speech_to_speech/api/openai_realtime/llm_proxy.py - Runtime configuration
src/speech_to_speech/api/openai_realtime/runtime_config.py
Speech Pipeline
- Pipeline orchestrator
src/speech_to_speech/s2s_pipeline.py - Voice detection
src/speech_to_speech/VAD/vad_handler.py - Transcription
src/speech_to_speech/STT/base_stt_handler.py - Transcription notifier
src/speech_to_speech/STT/transcription_notifier.py - Handler runtime
src/speech_to_speech/baseHandler.py
Model Backends
- Language model
src/speech_to_speech/LLM/chat.py - LLM backends
src/speech_to_speech/LLM - Speech synthesis
src/speech_to_speech/TTS - STT backends
src/speech_to_speech/STT
Pipeline Coordination
- Typed messages
src/speech_to_speech/pipeline/messages.py - Pipeline events
src/speech_to_speech/pipeline/events.py - Turn coordination
src/speech_to_speech/pipeline/speculative_turns.py - Cancellation scope
src/speech_to_speech/pipeline/cancel_scope.py - Transcript logging
src/speech_to_speech/pipeline/transcript_logging.py
Other components
- Realtime client
Connections 24
- Realtime client → Realtime server: connects via realtime
- Realtime server → WebSocket router: routes sessions
src/speech_to_speech/api/openai_realtime/server.py - WebSocket router → Realtime service: dispatches events
src/speech_to_speech/api/openai_realtime/websocket_router.py - WebSocket router → Session pipeline unit: claims session unit
src/speech_to_speech/api/openai_realtime/websocket_router.py - Pipeline orchestrator → Session pipeline unit: creates realtime unit
src/speech_to_speech/s2s_pipeline.py - Pipeline orchestrator → Voice detection: configures stage
src/speech_to_speech/s2s_pipeline.py - Pipeline orchestrator → Transcription: configures stage
src/speech_to_speech/s2s_pipeline.py - Pipeline orchestrator → Transcription notifier: configures notifier
src/speech_to_speech/s2s_pipeline.py - Pipeline orchestrator → Language model: configures stage
src/speech_to_speech/s2s_pipeline.py - Pipeline orchestrator → Speech synthesis: configures stage
src/speech_to_speech/s2s_pipeline.py - Voice detection → Transcription: passes audio
- Transcription → Language model: passes transcription
- Language model → Speech synthesis: passes response
- Pipeline orchestrator → Handler runtime: uses handler runtime
src/speech_to_speech/s2s_pipeline.py - Runtime configuration → Language model: holds chat state
src/speech_to_speech/api/openai_realtime/runtime_config.py - WebSocket router → LLM proxy: uses proxy
src/speech_to_speech/api/openai_realtime/websocket_router.py - Pipeline orchestrator → Typed messages: uses pipeline messages
src/speech_to_speech/s2s_pipeline.py - Handler runtime → Typed messages: processes typed inputs
src/speech_to_speech/baseHandler.py - Voice detection → Pipeline events: emits speech events
src/speech_to_speech/VAD/vad_handler.py - Realtime service → Turn coordination: tracks speculative turns
src/speech_to_speech/api/openai_realtime/service.py - Session pipeline unit → Cancellation scope: holds cancellation scope
src/speech_to_speech/api/openai_realtime/pipeline_unit.py - LLM backends → Language model: implements alternatives
- STT backends → Transcription: implements alternatives
src/speech_to_speech/STT/streaming_handler.py - Speech synthesis → Transcript logging: uses transcript logging
src/speech_to_speech/TTS/qwen3_tts_handler.py