How yolo-zyl/livetalking works
Project Type and Purpose
• This is a full-stack, real-time interactive streaming “digital human” platform.
• It ingests live audio, runs ASR → LLM chat → TTS/voice-clone → lip-sync & face/head animation (via ErNeRF, MuseTalk, or Wav2Lip) → streams synchronized audio/video back to the browser.
• Supports WebRTC & RTMP ingestion/output, multiple model backends, and custom video playback when silent.
High-Level Component Analysis
a. Front-End (web/)
• Static HTML/JS clients for: ASR demo, WebRTC push/pull, echo tests, chat UI.
• Uses recorder-core.js, WebRTC APIs, SRS SDK to publish/subcribe media channels.
b. Streaming Server (SRS)
• External containerized SRS instance (Docker) handling RTMP/WebRTC ingest and distribution.
c. Back-End Orchestrator (app.py + support scripts)
• app.py: central asyncio server receiving incoming media streams, routing through pipelines, servicing HTTP/WebSocket control endpoints (ports 8010 etc.).
• Dockerfile and deployment scripts.
d. Audio Processing
• baseasr.py / lipasr.py / museasr.py / nerfasr.py: wrappers around ASR engines (e.g., Whisper, wav2vec).
• llm/: ChatGPT/GPT/Qwen clients for generating bot replies.
• ttsreal.py / muse tts / edgetts integration for text-to-speech and voice cloning.
e. Video Generation
• ernerf/: Neural Radiance Field based talking head (freqencoder, gridencoder, raymarching, nerf_triplane).
• musetalk/: VAE+UNet-based face models + full-body compositing.
• wav2lip/: Conv-based lip-sync model + face detection.
f. Data & Models
• data/pretrained: PTH checkpoints for ErNeRF & custom ASR.
• muse tts & museasr pretrained checkpoints.
• wav2lip model weights.
g. External Services
• Hugging Face mirrors (HF_ENDPOINT override).
• Docker registry for SRS and base images.
Key Architectural Patterns
• Modular “plugin” pipelines for ASR, TTS, video engines: choose at runtime via configuration (data/custom_config.json).
• Asynchronous, streaming-first design: non-blocking I/O for media.
• Microservice-like separation: front-end client, media bus (SRS), compute back-end.
• Client Browser (HTML/JS)
• SRS Streaming Server
• Backend App Server
• ASR & LLM Pipeline
• TTS + Voice Clone
• Video Renderer Module (ErNeRF | MuseTalk | Wav2Lip)
• Data/Model Storage
• External HF Mirror
1. Browser → (WebRTC / RTMP) → SRS Ingest
2. SRS → (WebSocket / HTTP pull) → app.py
3. app.py → ASR → LLM → TTS → Voice Clone
4. app.py + Audio → Video Renderer → Synthesized Frames
5. app.py → (encode) → SRS Publish
6. Browser ← (WebRTC / RTMP) ← SRS
c. Asynchronous Steps:
d. Technologies:
• ASR: Whisper, wav2vec; LLM: ChatGPT/GPT/Qwen; TTS: edgetts; Rendering: PyTorch (CUDA), Pytorch3D, custom C++/CUDA kernels; Front-end: recorder-core.js, srs.sdk.js; Deployment: Docker.
e. Patterns:
• Plugin modules in separate subdirectories.
• Shared data models under data/.
• Dockerized components (SRS, app).