# kedreamix/linly-talker-stream architecture

> Architecture diagram of the GitHub repository kedreamix/linly-talker-stream, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kedreamix/linly-talker-stream
- Repository: https://github.com/kedreamix/linly-talker-stream

## Overview

1. **Overall shape**
   Linly-Talker-Stream is a split frontend/backend system for real-time digital-human conversation. The browser client handles user interaction and playback, while the Python backend orchestrates ASR, LLM, TTS, avatar rendering, and WebRTC streaming.

2. **Frontend boundary**
   The `web/` app is a Vue 3 + Vite client. Its main responsibilities are browser media capture, speech-recognition UI, WebRTC session setup, settings/config selection, and video/audio playback. The frontend is the user-facing edge of the system, not where model inference happens.

3. **Backend boundary**
   The backend lives under `src/server/` and is the central coordination layer. It exposes the HTTP/WebRTC API surface, keeps runtime state, routes requests to the multimodal pipeline, and returns synthesized audio/video or streaming session negotiation data.

4. **Realtime transport**
   WebRTC is the key transport boundary. `POST /offer` performs SDP negotiation, and the backend uses `aiortc`/`aiohttp`-style server plumbing to stream generated media back to the browser with low latency. This is the main mechanism that turns the system from request/response into full-duplex interaction.

5. **Conversation pipeline**
   The core data flow is: browser microphone input → ASR → text generation via LLM → speech synthesis via TTS → avatar driving/rendering → streamed audio/video back to the browser. The repository explicitly supports both browser-side speech recognition and server-side ASR paths, which affects where the first transcription boundary sits.

6. **ASR subsystem**
   `src/asr/` contains pluggable speech-to-text engines. The concrete engines shown are Whisper and FunASR, selected through a factory layer. This subsystem is independent from the avatar engines and can be swapped based on deployment or language requirements.

7. **LLM subsystem**
   `src/llm/` is a thin adapter layer around text-generation backends. The repo includes an OpenAI-style engine wrapper, and the README indicates DashScope/Qwen-plus as the default service. The LLM is treated as a service boundary, not a local model bundle.

8. **TTS subsystem**
   `src/tts/` provides multiple synthesis backends: Azure, Edge, CosyVoice, Fish, XTTS, Tencent, Doubao, SoVITS, and others. This makes speech generation another pluggable boundary, with providers ranging from cloud APIs to local models.

9. **Avatar engines**
   `src/avatars/` is the largest subsystem and contains multiple mutually replaceable avatar pipelines: `wav2lip`, `musetalk`, `ernerf`, `talkinggaussian`, plus additional lightweight variants. Each avatar package owns its own audio stream handler, model logic, preprocessing, and asset dependencies. The backend chooses one based on `model.type` in config.

10. **GPU-heavy submodules**
   The 3D/avatar-rendering paths include custom CUDA and rasterization code. `talkinggaussian` pulls in `diff-gaussian-rasterization`, `simple-knn`, and `gridencoder`; `ernerf` also includes CUDA-based encoders and raymarching. These are separate build/install boundaries and are the most infrastructure-sensitive part of the repo.

11. **Preprocessing and asset preparation**
   Several avatar packages include data-prep pipelines for face detection, face parsing, tracking, audio feature extraction, and avatar generation. These are offline or startup-time preparation steps that feed the runtime avatar engines with weights, landmarks, meshes, or extracted features.

12. **Configuration and startup**
   `config/*.yaml` drives the whole runtime: server port, SSL, ASR mode, avatar type, TTS engine, and LLM settings. `src/config/` loads and validates these settings, while `scripts/` handles environment setup, module-specific installs, SSL certificate creation, and starting backend/frontend together.

13. **External dependencies and deployment constraints**
   The system depends on browser media APIs, HTTPS for microphone access in remote use, and external model/service credentials such as `DASHSCOPE_API_KEY`. Because of the modular avatar packages and GPU extensions, installation is intentionally split so deployments only pull in the selected avatar stack.

14. **Useful diagram boundaries**
   For an architecture diagram, the main boxes should be: browser UI, WebRTC transport, backend orchestration, ASR/LLM/TTS services, avatar engine variants, and optional GPU/CUDA submodules. The clearest arrows are media/input flow from browser to backend, text flow through ASR/LLM/TTS, and synthesized media flow back over WebRTC.

Main entry points: [`src/server/routes`](https://github.com/kedreamix/linly-talker-stream/tree/main/src/server/routes) (API Routes), [`src/avatars/factory.py`](https://github.com/kedreamix/linly-talker-stream/blob/main/src/avatars/factory.py) (Avatar Factory), [`src/config/loader.py`](https://github.com/kedreamix/linly-talker-stream/blob/main/src/config/loader.py) (Config Loader), [`web`](https://github.com/kedreamix/linly-talker-stream/tree/main/web) (Web App)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Browser UI"]
  node_web_app["Web App<br/>Vue 3 client"]
  node_web_webrtc["WebRTC Hook<br/>media/session<br/>[useWebRTC.js]"]
  node_web_speech["Speech Hook<br/>recognition/ui"]
  node_web_video["Video Player<br/>playback<br/>[VideoPlayer.vue]"]
end

subgraph group_backend["Python Backend"]
  node_server_app["Server App<br/>aiohttp/aiortc app<br/>[app.py]"]
  node_server_routes["API Routes<br/>HTTP/WebRTC endpoints"]
  node_server_state["Runtime State<br/>session state<br/>[state.py]"]
  node_webrtc_utils["WebRTC Utils<br/>transport helpers<br/>[webrtc.py]"]
end

subgraph group_speech["Speech Services"]
  node_asr_factory["ASR Factory<br/>engine selector<br/>[factory.py]"]
  node_asr_engines["ASR Engines<br/>speech-to-text"]
  node_llm_service["LLM Service<br/>generation adapter<br/>[service.py]"]
  node_llm_openai["OpenAI Adapter<br/>chat backend<br/>[openai.py]"]
  node_tts_factory["TTS Factory<br/>engine selector<br/>[factory.py]"]
  node_tts_engines["TTS Engines<br/>speech synthesis"]
end

subgraph group_avatars["Avatar Engines"]
  node_avatar_factory["Avatar Factory<br/>model selector<br/>[factory.py]"]
  node_wav2lip["Wav2Lip<br/>avatar pipeline"]
  node_musetalk["MuseTalk<br/>avatar pipeline"]
  node_ernerf["ER-NeRF<br/>3D avatar pipeline"]
  node_tg_avatar["Talking Gaussian<br/>3D avatar pipeline"]
  node_ultralight["Ultralight<br/>light avatar"]
end

subgraph group_gpu["GPU Submodules"]
  node_cuda_exts["CUDA Extensions<br/>rasterization/encoders"]
end

subgraph group_startup["Startup & Config"]
  node_config_loader["Config Loader<br/>yaml/schema<br/>[loader.py]"]
  node_config_yaml["Config Files<br/>deployment config<br/>[config.yaml]"]
  node_scripts["Launch Scripts<br/>setup/startup"]
end

node_web_app -->|"setup session"| node_web_webrtc
node_web_app -->|"capture speech"| node_web_speech
node_web_app -->|"play media"| node_web_video
node_web_webrtc -->|"offer/answer"| node_server_routes
node_server_app -->|"mount routes"| node_server_routes
node_server_app -->|"share state"| node_server_state
node_server_routes -->|"transport"| node_webrtc_utils
node_server_routes -->|"transcribe"| node_asr_factory
node_server_routes -->|"generate reply"| node_llm_service
node_server_routes -->|"synthesize voice"| node_tts_factory
node_server_routes -->|"drive avatar"| node_avatar_factory
node_asr_factory -->|"whisper/funasr"| node_asr_engines
node_llm_service -->|"model adapter"| node_llm_openai
node_tts_factory -->|"provider pick"| node_tts_engines
node_avatar_factory -->|"select"| node_wav2lip
node_avatar_factory -->|"select"| node_musetalk
node_avatar_factory -->|"select"| node_ernerf
node_avatar_factory -->|"select"| node_tg_avatar
node_avatar_factory -->|"select"| node_ultralight
node_ernerf -->|"uses"| node_cuda_exts
node_tg_avatar -->|"uses"| node_cuda_exts
node_config_yaml -->|"load"| node_config_loader
node_config_loader -->|"configure runtime"| node_server_app
node_config_loader -->|"model.type"| node_avatar_factory
node_config_loader -->|"asr mode"| node_asr_factory
node_config_loader -->|"tts engine"| node_tts_factory
node_config_loader -->|"llm config"| node_llm_service
node_scripts -->|"seed settings"| node_config_yaml
node_scripts -->|"start backend"| node_server_app
node_scripts -->|"start frontend"| node_web_app

click node_web_app "https://github.com/kedreamix/linly-talker-stream/tree/main/web"
click node_web_webrtc "https://github.com/kedreamix/linly-talker-stream/blob/main/web/src/composables/useWebRTC.js"
click node_web_speech "https://github.com/kedreamix/linly-talker-stream/blob/main/web/src/composables/useSpeechRecognition.js"
click node_web_video "https://github.com/kedreamix/linly-talker-stream/blob/main/web/src/components/VideoPlayer.vue"
click node_server_app "https://github.com/kedreamix/linly-talker-stream/blob/main/src/server/app.py"
click node_server_routes "https://github.com/kedreamix/linly-talker-stream/tree/main/src/server/routes"
click node_server_state "https://github.com/kedreamix/linly-talker-stream/blob/main/src/server/state.py"
click node_webrtc_utils "https://github.com/kedreamix/linly-talker-stream/blob/main/src/utils/webrtc.py"
click node_asr_factory "https://github.com/kedreamix/linly-talker-stream/blob/main/src/asr/factory.py"
click node_asr_engines "https://github.com/kedreamix/linly-talker-stream/tree/main/src/asr/engines"
click node_llm_service "https://github.com/kedreamix/linly-talker-stream/blob/main/src/llm/service.py"
click node_llm_openai "https://github.com/kedreamix/linly-talker-stream/blob/main/src/llm/engines/openai.py"
click node_tts_factory "https://github.com/kedreamix/linly-talker-stream/blob/main/src/tts/factory.py"
click node_tts_engines "https://github.com/kedreamix/linly-talker-stream/tree/main/src/tts/engines"
click node_avatar_factory "https://github.com/kedreamix/linly-talker-stream/blob/main/src/avatars/factory.py"
click node_wav2lip "https://github.com/kedreamix/linly-talker-stream/tree/main/src/avatars/wav2lip"
click node_musetalk "https://github.com/kedreamix/linly-talker-stream/tree/main/src/avatars/musetalk"
click node_ernerf "https://github.com/kedreamix/linly-talker-stream/tree/main/src/avatars/ernerf"
click node_tg_avatar "https://github.com/kedreamix/linly-talker-stream/tree/main/src/avatars/talkinggaussian"
click node_ultralight "https://github.com/kedreamix/linly-talker-stream/tree/main/src/avatars/ultralight"
click node_cuda_exts "https://github.com/kedreamix/linly-talker-stream/tree/main/src/avatars/talkinggaussian/submodules"
click node_config_loader "https://github.com/kedreamix/linly-talker-stream/blob/main/src/config/loader.py"
click node_config_yaml "https://github.com/kedreamix/linly-talker-stream/blob/main/config/config.yaml"
click node_scripts "https://github.com/kedreamix/linly-talker-stream/tree/main/scripts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_web_app,node_web_webrtc,node_web_speech,node_web_video toneBlue
class node_server_app,node_server_routes,node_server_state,node_webrtc_utils toneAmber
class node_asr_factory,node_asr_engines,node_llm_service,node_llm_openai,node_tts_factory,node_tts_engines toneMint
class node_avatar_factory,node_wav2lip,node_musetalk,node_ernerf,node_tg_avatar,node_ultralight toneRose
class node_cuda_exts toneIndigo
class node_config_loader,node_config_yaml,node_scripts toneTeal
```

## Components

### Browser UI

Vue client for interaction, media capture, and playback.

- [Web App](https://github.com/kedreamix/linly-talker-stream/tree/main/web): Vue 3 client · Browser-side interface for interaction, settings, recording, and playback. · `web`
- [WebRTC Hook](https://github.com/kedreamix/linly-talker-stream/blob/main/web/src/composables/useWebRTC.js): media/session · Browser composable for session setup and realtime transport. · `web/src/composables/useWebRTC.js`
- [Speech Hook](https://github.com/kedreamix/linly-talker-stream/blob/main/web/src/composables/useSpeechRecognition.js): recognition/ui · Browser speech-recognition integration for user input. · `web/src/composables/useSpeechRecognition.js`
- [Video Player](https://github.com/kedreamix/linly-talker-stream/blob/main/web/src/components/VideoPlayer.vue): playback · Client-side audio/video rendering surface. · `web/src/components/VideoPlayer.vue`

### Python Backend

Server-side orchestration, routing, and runtime state.

- [Server App](https://github.com/kedreamix/linly-talker-stream/blob/main/src/server/app.py): aiohttp/aiortc app · Backend application entry that assembles API routes and runtime services. · `src/server/app.py`
- [API Routes](https://github.com/kedreamix/linly-talker-stream/tree/main/src/server/routes): HTTP/WebRTC endpoints · HTTP surface for chat, audio, video, health, and SDP negotiation. · `src/server/routes`
- [Runtime State](https://github.com/kedreamix/linly-talker-stream/blob/main/src/server/state.py): session state · Shared backend state for active conversations and stream coordination. · `src/server/state.py`
- [WebRTC Utils](https://github.com/kedreamix/linly-talker-stream/blob/main/src/utils/webrtc.py): transport helpers · Helpers for session negotiation and media streaming plumbing. · `src/utils/webrtc.py`

### Speech Services

Pluggable ASR, LLM, and TTS layers.

- [ASR Factory](https://github.com/kedreamix/linly-talker-stream/blob/main/src/asr/factory.py): engine selector · Chooses the active speech-to-text backend. · `src/asr/factory.py`
- [ASR Engines](https://github.com/kedreamix/linly-talker-stream/tree/main/src/asr/engines): speech-to-text · Whisper and FunASR implementations behind a common interface. · `src/asr/engines`
- [LLM Service](https://github.com/kedreamix/linly-talker-stream/blob/main/src/llm/service.py): generation adapter · Text-generation service boundary used by the conversation loop. · `src/llm/service.py`
- [OpenAI Adapter](https://github.com/kedreamix/linly-talker-stream/blob/main/src/llm/engines/openai.py): chat backend · OpenAI-style LLM engine wrapper, including service-hosted models. · `src/llm/engines/openai.py`
- [TTS Factory](https://github.com/kedreamix/linly-talker-stream/blob/main/src/tts/factory.py): engine selector · Chooses the active text-to-speech backend. · `src/tts/factory.py`
- [TTS Engines](https://github.com/kedreamix/linly-talker-stream/tree/main/src/tts/engines): speech synthesis · Pluggable synthesis backends such as Azure, Edge, CosyVoice, XTTS, and others. · `src/tts/engines`

### Avatar Engines

Replaceable avatar pipelines and renderers.

- [Avatar Factory](https://github.com/kedreamix/linly-talker-stream/blob/main/src/avatars/factory.py): model selector · Selects the avatar pipeline from configuration. · `src/avatars/factory.py`
- [Wav2Lip](https://github.com/kedreamix/linly-talker-stream/tree/main/src/avatars/wav2lip): avatar pipeline · Lip-sync avatar path for audio-driven facial animation. · `src/avatars/wav2lip`
- [MuseTalk](https://github.com/kedreamix/linly-talker-stream/tree/main/src/avatars/musetalk): avatar pipeline · Talking-head avatar path with its own preprocessing and runtime handlers. · `src/avatars/musetalk`
- [ER-NeRF](https://github.com/kedreamix/linly-talker-stream/tree/main/src/avatars/ernerf): 3D avatar pipeline · Neural rendering avatar path with preprocessing and GPU-heavy components. · `src/avatars/ernerf`
- [Talking Gaussian](https://github.com/kedreamix/linly-talker-stream/tree/main/src/avatars/talkinggaussian): 3D avatar pipeline · Gaussian-splatting avatar path for realtime rendering. · `src/avatars/talkinggaussian`
- [Ultralight](https://github.com/kedreamix/linly-talker-stream/tree/main/src/avatars/ultralight): light avatar · Lightweight avatar path for faster or simpler deployments. · `src/avatars/ultralight`

### GPU Submodules

CUDA and rasterization extensions used by 3D avatar paths.

- [CUDA Extensions](https://github.com/kedreamix/linly-talker-stream/tree/main/src/avatars/talkinggaussian/submodules): rasterization/encoders · Custom GPU submodules for Gaussian rendering and neural encoding. · `src/avatars/talkinggaussian/submodules`

### Startup & Config

Config loading and launch scripts that wire the system together.

- [Config Loader](https://github.com/kedreamix/linly-talker-stream/blob/main/src/config/loader.py): yaml/schema · Loads and validates runtime configuration. · `src/config/loader.py`
- [Config Files](https://github.com/kedreamix/linly-talker-stream/blob/main/config/config.yaml): deployment config · YAML settings for server, ASR, avatar, TTS, and LLM selection. · `config/config.yaml`
- [Launch Scripts](https://github.com/kedreamix/linly-talker-stream/tree/main/scripts): setup/startup · Shell scripts for setup, SSL creation, and starting backend/frontend. · `scripts`

## Connections

- Web App → WebRTC Hook: setup session · The UI opens realtime media sessions through the WebRTC composable.
- Web App → Speech Hook: capture speech · The UI exposes browser speech-recognition input.
- Web App → Video Player: play media · The UI renders streamed audio/video for the conversation.
- WebRTC Hook → API Routes: offer/answer · Browser negotiation reaches the backend WebRTC route.
- Server App → API Routes: mount routes · The backend app assembles the HTTP and WebRTC endpoints.
- Server App → Runtime State: share state · The backend keeps conversation and stream state in a shared store.
- API Routes → WebRTC Utils: transport · Routes delegate session and media handling to transport helpers.
- API Routes → ASR Factory: transcribe · Incoming audio or text is handed to the ASR selection layer.
- API Routes → LLM Service: generate reply · Chat requests are routed to the LLM service boundary.
- API Routes → TTS Factory: synthesize voice · Reply text is forwarded to speech synthesis.
- API Routes → Avatar Factory: drive avatar · Synthesized speech is handed to the avatar selector.
- ASR Factory → ASR Engines: whisper/funasr · The factory chooses among concrete ASR engines.
- LLM Service → OpenAI Adapter: model adapter · The service wraps an OpenAI-style generation backend.
- TTS Factory → TTS Engines: provider pick · The factory selects one of the pluggable TTS providers.
- Avatar Factory → Wav2Lip: select · One configurable avatar path is Wav2Lip.
- Avatar Factory → MuseTalk: select · One configurable avatar path is MuseTalk.
- Avatar Factory → ER-NeRF: select · One configurable avatar path is ER-NeRF.
- Avatar Factory → Talking Gaussian: select · One configurable avatar path is Talking Gaussian.
- Avatar Factory → Ultralight: select · One configurable avatar path is the lightweight avatar stack.
- ER-NeRF → CUDA Extensions: uses · ER-NeRF depends on custom GPU encoders and rasterization pieces.
- Talking Gaussian → CUDA Extensions: uses · Talking Gaussian depends on CUDA and rasterization submodules.
- Config Files → Config Loader: load · YAML files are parsed into validated runtime settings.
- Config Loader → Server App: configure runtime · Loaded settings determine server behavior and model selection.
- Config Loader → Avatar Factory: model.type · Configuration picks the active avatar engine.
- Config Loader → ASR Factory: asr mode · Configuration picks the active ASR backend.
- Config Loader → TTS Factory: tts engine · Configuration picks the active TTS backend.
- Config Loader → LLM Service: llm config · Configuration supplies the conversation model settings.
- Launch Scripts → Config Files: seed settings · Setup scripts populate and use configuration files.
- Launch Scripts → Server App: start backend · Launch scripts boot the Python server.
- Launch Scripts → Web App: start frontend · Launch scripts boot the browser client.