# huggingface/speech-to-speech architecture

> Architecture diagram of the GitHub repository huggingface/speech-to-speech, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/huggingface/speech-to-speech
- Repository: https://github.com/huggingface/speech-to-speech

## Overview

Speech To Speech is a modular, low-latency voice-agent runtime. Its principal workflow takes client audio through voice activity detection, transcription, language-model response generation, and speech synthesis, returning realtime events and audio to the client. The repository also exposes OpenAI Realtime-compatible WebSocket and WebRTC sessions, supports interchangeable model backends, and includes internal turn, cancellation, and typed-message coordination. Edges follow README statements and shown source references; unsampled backend and transport details are represented only at the directory or subsystem level.

Main entry points: [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (Pipeline orchestrator), [`src/speech_to_speech/LLM/chat.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat.py) (Language model), [`src/speech_to_speech/api/openai_realtime/websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py) (WebSocket router), [`src/speech_to_speech/STT/base_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/base_stt_handler.py) (Transcription)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_realtime["Realtime API"]
  node_server["Realtime server<br/>[server.py]"]
  node_router["WebSocket router"]
  node_session_service["Realtime service<br/>[service.py]"]
  node_pipeline_unit["Session pipeline unit<br/>[pipeline_unit.py]"]
  node_llm_proxy["LLM proxy<br/>[llm_proxy.py]"]
  node_runtime_config["Runtime configuration<br/>[runtime_config.py]"]
end

subgraph group_speech["Speech Pipeline"]
  node_pipeline["Pipeline orchestrator<br/>[s2s_pipeline.py]"]
  node_vad["Voice detection<br/>[vad_handler.py]"]
  node_stt["Transcription"]
  node_notifier["Transcription notifier"]
  node_base_handler["Handler runtime<br/>[baseHandler.py]"]
end

subgraph group_models["Model Backends"]
  node_llm["Language model<br/>[chat.py]"]
  node_llm_backends["LLM backends"]
  node_tts["Speech synthesis"]
  node_stt_backends["STT backends"]
end

subgraph group_coordination["Pipeline Coordination"]
  node_messages["Typed messages<br/>[messages.py]"]
  node_events["Pipeline events<br/>[events.py]"]
  node_turns["Turn coordination"]
  node_cancellation["Cancellation scope<br/>[cancel_scope.py]"]
  node_transcript_logging["Transcript logging"]
end

node_client(("Realtime client"))

node_client -->|"connects via realtime"| node_server
node_server -->|"routes sessions"| node_router
node_router -->|"dispatches events"| node_session_service
node_router -->|"claims session unit"| node_pipeline_unit
node_pipeline -->|"creates realtime unit"| node_pipeline_unit
node_pipeline -->|"configures stage"| node_vad
node_pipeline -->|"configures stage"| node_stt
node_pipeline -->|"configures notifier"| node_notifier
node_pipeline -->|"configures stage"| node_llm
node_pipeline -->|"configures stage"| node_tts
node_vad -->|"passes audio"| node_stt
node_stt -->|"passes transcription"| node_llm
node_llm -->|"passes response"| node_tts
node_pipeline -->|"uses handler runtime"| node_base_handler
node_runtime_config -->|"holds chat state"| node_llm
node_router -->|"uses proxy"| node_llm_proxy
node_pipeline -->|"uses pipeline messages"| node_messages
node_base_handler -->|"processes typed inputs"| node_messages
node_vad -->|"emits speech events"| node_events
node_session_service -->|"tracks speculative turns"| node_turns
node_pipeline_unit -->|"holds cancellation scope"| node_cancellation
node_llm_backends -.->|"implements alternatives"| node_llm
node_stt_backends -.->|"implements alternatives"| node_stt
node_tts -->|"uses transcript logging"| node_transcript_logging

click node_server "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/server.py"
click node_router "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py"
click node_session_service "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/service.py"
click node_pipeline_unit "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/pipeline_unit.py"
click node_llm_proxy "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/llm_proxy.py"
click node_runtime_config "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py"
click node_pipeline "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py"
click node_vad "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py"
click node_stt "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/base_stt_handler.py"
click node_notifier "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/transcription_notifier.py"
click node_llm "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat.py"
click node_llm_backends "https://github.com/huggingface/speech-to-speech/tree/main/src/speech_to_speech/LLM"
click node_tts "https://github.com/huggingface/speech-to-speech/tree/main/src/speech_to_speech/TTS"
click node_stt_backends "https://github.com/huggingface/speech-to-speech/tree/main/src/speech_to_speech/STT"
click node_base_handler "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/baseHandler.py"
click node_messages "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py"
click node_events "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/events.py"
click node_turns "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py"
click node_cancellation "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/cancel_scope.py"
click node_transcript_logging "https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/transcript_logging.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_server,node_router,node_session_service,node_pipeline_unit,node_llm_proxy,node_runtime_config,node_client toneBlue
class node_pipeline,node_vad,node_stt,node_notifier,node_base_handler toneAmber
class node_llm,node_llm_backends,node_tts,node_stt_backends toneMint
class node_messages,node_events,node_turns,node_cancellation,node_transcript_logging toneRose
```

## Components

### Realtime API

- [Realtime server](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/server.py): `src/speech_to_speech/api/openai_realtime/server.py`
- [WebSocket router](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py): `src/speech_to_speech/api/openai_realtime/websocket_router.py`
- [Realtime service](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/service.py): `src/speech_to_speech/api/openai_realtime/service.py`
- [Session pipeline unit](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/pipeline_unit.py): `src/speech_to_speech/api/openai_realtime/pipeline_unit.py`
- [LLM proxy](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/llm_proxy.py): `src/speech_to_speech/api/openai_realtime/llm_proxy.py`
- [Runtime configuration](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py): `src/speech_to_speech/api/openai_realtime/runtime_config.py`

### Speech Pipeline

- [Pipeline orchestrator](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py): `src/speech_to_speech/s2s_pipeline.py`
- [Voice detection](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py): `src/speech_to_speech/VAD/vad_handler.py`
- [Transcription](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/base_stt_handler.py): `src/speech_to_speech/STT/base_stt_handler.py`
- [Transcription notifier](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/transcription_notifier.py): `src/speech_to_speech/STT/transcription_notifier.py`
- [Handler runtime](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/baseHandler.py): `src/speech_to_speech/baseHandler.py`

### Model Backends

- [Language model](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat.py): `src/speech_to_speech/LLM/chat.py`
- [LLM backends](https://github.com/huggingface/speech-to-speech/tree/main/src/speech_to_speech/LLM): `src/speech_to_speech/LLM`
- [Speech synthesis](https://github.com/huggingface/speech-to-speech/tree/main/src/speech_to_speech/TTS): `src/speech_to_speech/TTS`
- [STT backends](https://github.com/huggingface/speech-to-speech/tree/main/src/speech_to_speech/STT): `src/speech_to_speech/STT`

### Pipeline Coordination

- [Typed messages](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py): `src/speech_to_speech/pipeline/messages.py`
- [Pipeline events](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/events.py): `src/speech_to_speech/pipeline/events.py`
- [Turn coordination](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py): `src/speech_to_speech/pipeline/speculative_turns.py`
- [Cancellation scope](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/cancel_scope.py): `src/speech_to_speech/pipeline/cancel_scope.py`
- [Transcript logging](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/transcript_logging.py): `src/speech_to_speech/pipeline/transcript_logging.py`

### Other components

- **Realtime client**

## Connections

- Realtime client → Realtime server: connects via realtime
- Realtime server → WebSocket router: routes sessions (evidence: `src/speech_to_speech/api/openai_realtime/server.py`)
- WebSocket router → Realtime service: dispatches events (evidence: `src/speech_to_speech/api/openai_realtime/websocket_router.py`)
- WebSocket router → Session pipeline unit: claims session unit (evidence: `src/speech_to_speech/api/openai_realtime/websocket_router.py`)
- Pipeline orchestrator → Session pipeline unit: creates realtime unit (evidence: `src/speech_to_speech/s2s_pipeline.py`)
- Pipeline orchestrator → Voice detection: configures stage (evidence: `src/speech_to_speech/s2s_pipeline.py`)
- Pipeline orchestrator → Transcription: configures stage (evidence: `src/speech_to_speech/s2s_pipeline.py`)
- Pipeline orchestrator → Transcription notifier: configures notifier (evidence: `src/speech_to_speech/s2s_pipeline.py`)
- Pipeline orchestrator → Language model: configures stage (evidence: `src/speech_to_speech/s2s_pipeline.py`)
- Pipeline orchestrator → Speech synthesis: configures stage (evidence: `src/speech_to_speech/s2s_pipeline.py`)
- Voice detection → Transcription: passes audio
- Transcription → Language model: passes transcription
- Language model → Speech synthesis: passes response
- Pipeline orchestrator → Handler runtime: uses handler runtime (evidence: `src/speech_to_speech/s2s_pipeline.py`)
- Runtime configuration → Language model: holds chat state (evidence: `src/speech_to_speech/api/openai_realtime/runtime_config.py`)
- WebSocket router → LLM proxy: uses proxy (evidence: `src/speech_to_speech/api/openai_realtime/websocket_router.py`)
- Pipeline orchestrator → Typed messages: uses pipeline messages (evidence: `src/speech_to_speech/s2s_pipeline.py`)
- Handler runtime → Typed messages: processes typed inputs (evidence: `src/speech_to_speech/baseHandler.py`)
- Voice detection → Pipeline events: emits speech events (evidence: `src/speech_to_speech/VAD/vad_handler.py`)
- Realtime service → Turn coordination: tracks speculative turns (evidence: `src/speech_to_speech/api/openai_realtime/service.py`)
- Session pipeline unit → Cancellation scope: holds cancellation scope (evidence: `src/speech_to_speech/api/openai_realtime/pipeline_unit.py`)
- LLM backends → Language model: implements alternatives
- STT backends → Transcription: implements alternatives (evidence: `src/speech_to_speech/STT/streaming_handler.py`)
- Speech synthesis → Transcript logging: uses transcript logging (evidence: `src/speech_to_speech/TTS/qwen3_tts_handler.py`)
