How mu-l/livetalking works
Project type and purpose
• This is a full-stack, real-time interactive “digital human” streaming system.
• It exposes a Python-based backend (app.py + various model modules) that orchestrates ASR → language (LLM) → TTS → avatar rendering pipelines, and a web-based frontend for WebRTC/RTMP streaming of audio/video.
File structure
• Top-level entry points:
• app.py / webrtc.py – main server processes (HTTP/WebSocket/WebRTC signaling, transport selection)
• Dockerfile – containerization
• requirements.txt – Python dependencies (PyTorch, etc.)
• Service modules (all under project root):
• ASR: baseasr.py, hubertasr.py, lipasr.py, lipreal.py, lightreal.py, museasr.py, musereal.py
• Whisper ASR: musetalk/whisper/...
• TTS: ttsreal.py (plus fish-speech, Tencent, 豆包, etc.)
• LLM: llm.py (conversation management, optional chat model)
• Avatar rendering: wav2lip/, ultralight/, musetalk/ (models/unet.py, vae.py, simple_musetalk.py), ernerf branch (in separate git branch)
• Utilities: logger.py, muse/utils, wav2lip/face_detection, museTalk/utils, ultralight/audio2feature & genavatar
• Frontend: web/ (HTML/JS pages for ASR, chat, webrtc, RTMP push, virtual camera)
• Data & static assets: data/avatars, models/ (place for downloaded .pth), assets/ (demo, FAQ)
README for additional insights
• Features: multi-model support (wav2lip256, musetalk, Ultralight, ernerf), voice cloning, interruptible speech, full-body splicing, action choreography, multi-concurrency
• Transports: WebRTC, RTMP, virtual camera output
• External services: optional SRS server, HuggingFace mirror, cloud-hosted Docker images
Main components
a. Clients:
• Browser UI (web/*.html + JS)
• Custom client (native executable)
b. Signaling & Transport Layer:
• HTTP / WebSocket API (app.py)
• WebRTC signaling & media connections (webrtc.py)
• RTMP/WebSocket push (SRS or built-in)
c. Backend Orchestrator:
• Request dispatcher (audio/video ↔ pipelines)
• Module registry (chooses ASR, TTS, avatar model per request)
• Concurrency manager (asyncio / multi-thread)
d. ASR Subsystem:
• Whisper or custom ASR modules
• API: audio frames in → text out
e. Language / Dialogue Subsystem:
• llm.py (optional LLM/chat model)
• Handles interruptions, context management
f. TTS Subsystem:
• ttsreal.py + plugins (fish-speech, Tencent, 豆包)
• audio synthesis: text in → waveform out
g. Avatar Rendering Subsystem:
• wav2lip module (lip sync)
• musetalk (UNet + VAE)
• ultralight (lightweight digital human)
• ernerf (NeRF-based multi-view)
• Preprocessing (face detection, segmentation)
• Model loading & inference
h. Media Mixer & Encoder:
• Composes audio + video frames
• Splices full-body, overlays custom “idle” animations when silent
• Virtual camera output plugin
i. Transport Outbound:
• WebRTC media streams back to client
• RTMP push to SRS or CDN
j. Data & Configuration:
• data/avatars (avatar assets)
• data/custom_config.json
• models/*.pth
k. External Dependencies:
• SRS streaming server (optional)
• HuggingFace model hub / mirrors
• Docker registry
Relationships & data flows
1) Client → HTTP/WebSocket (start session, choose transport & model)
2) Client → audio media → Signaling/Transport
3) Transport layer → Orchestrator → ASR → text transcript
4) Transcript → (optional) LLM → dialog text
5) Text → TTS → audio waveform
6) Audio + silent intervals → Avatar renderer → video frames
7) Media Mixer → combined A/V stream
8) Stream → Transport outbound → Client
Architectural patterns & principles
• Modular “plugin” architecture for ASR/TTS/avatar engines
• Pipeline pattern for sequential media processing
• Async / concurrency to handle multiple concurrent sessions without GPU memory blow-up
• Separation of concerns: signaling, inference, media encoding, transport
Essential elements in detail
• Concurrency manager interacting with each inference component
• Realtime constraints (25+ fps target, CPU vs GPU bottlenecks)