kedreamix/linly-talker-stream

1. Overall shape

Linly-Talker-Stream is a split frontend/backend system for real-time digital-human conversation. The browser client handles user interaction and playback, while the Python backend orchestrates ASR, LLM, TTS, avatar rendering, and WebRTC streaming.

2. Frontend boundary

The web/ app is a Vue 3 + Vite client. Its main responsibilities are browser media capture, speech-recognition UI, WebRTC session setup, settings/config selection, and video/audio playback. The frontend is the user-facing edge of the system, not where model inference happens.

3. Backend boundary

The backend lives under src/server/ and is the central coordination layer. It exposes the HTTP/WebRTC API surface, keeps runtime state, routes requests to the multimodal pipeline, and returns synthesized audio/video or streaming session negotiation data.

4. Realtime transport

WebRTC is the key transport boundary. POST /offer performs SDP negotiation, and the backend uses aiortc/aiohttp-style server plumbing to stream generated media back to the browser with low latency. This is the main mechanism that turns the system from request/response into full-duplex interaction.

5. Conversation pipeline

The core data flow is: browser microphone input → ASR → text generation via LLM → speech synthesis via TTS → avatar driving/rendering → streamed audio/video back to the browser. The repository explicitly supports both browser-side speech recognition and server-side ASR paths, which affects where the first transcription boundary sits.

6. ASR subsystem

src/asr/ contains pluggable speech-to-text engines. The concrete engines shown are Whisper and FunASR, selected through a factory layer. This subsystem is independent from the avatar engines and can be swapped based on deployment or language requirements.

7. LLM subsystem

src/llm/ is a thin adapter layer around text-generation backends. The repo includes an OpenAI-style engine wrapper, and the README indicates DashScope/Qwen-plus as the default service. The LLM is treated as a service boundary, not a local model bundle.

8. TTS subsystem

src/tts/ provides multiple synthesis backends: Azure, Edge, CosyVoice, Fish, XTTS, Tencent, Doubao, SoVITS, and others. This makes speech generation another pluggable boundary, with providers ranging from cloud APIs to local models.

9. Avatar engines

src/avatars/ is the largest subsystem and contains multiple mutually replaceable avatar pipelines: wav2lip, musetalk, ernerf, talkinggaussian, plus additional lightweight variants. Each avatar package owns its own audio stream handler, model logic, preprocessing, and asset dependencies. The backend chooses one based on model.type in config.

10. GPU-heavy submodules

The 3D/avatar-rendering paths include custom CUDA and rasterization code. talkinggaussian pulls in diff-gaussian-rasterization, simple-knn, and gridencoder; ernerf also includes CUDA-based encoders and raymarching. These are separate build/install boundaries and are the most infrastructure-sensitive part of the repo.

11. Preprocessing and asset preparation

Several avatar packages include data-prep pipelines for face detection, face parsing, tracking, audio feature extraction, and avatar generation. These are offline or startup-time preparation steps that feed the runtime avatar engines with weights, landmarks, meshes, or extracted features.

12. Configuration and startup

config/*.yaml drives the whole runtime: server port, SSL, ASR mode, avatar type, TTS engine, and LLM settings. src/config/ loads and validates these settings, while scripts/ handles environment setup, module-specific installs, SSL certificate creation, and starting backend/frontend together.

13. External dependencies and deployment constraints

The system depends on browser media APIs, HTTPS for microphone access in remote use, and external model/service credentials such as DASHSCOPE_API_KEY. Because of the modular avatar packages and GPU extensions, installation is intentionally split so deployments only pull in the selected avatar stack.

14. Useful diagram boundaries

For an architecture diagram, the main boxes should be: browser UI, WebRTC transport, backend orchestration, ASR/LLM/TTS services, avatar engine variants, and optional GPU/CUDA submodules. The clearest arrows are media/input flow from browser to backend, text flow through ASR/LLM/TTS, and synthesized media flow back over WebRTC.