How opentalker/sadtalker works
SadTalker turns a portrait and driving audio into a talking-head video, with reference motion, still/full-image preprocessing, pose controls, and optional enhancement. The main flow runs through the Gradio interface or inference entry point: prepare the portrait, derive audio-conditioned coefficients, assemble render data, animate, and save video. Model internals support 3D face reconstruction, audio-to-expression/pose prediction, and neural rendering. Some orchestration code is absent from the supplied tree or excerpts, so those links are omitted or marked unverified rather than inferred as direct calls.
Main entry points: animate.py Animation Orchestrator · gradio_demo.py Gradio Demo · generate_batch.py Audio Features · preprocess.py Portrait Prep
Components 15
User Interfaces
- Gradio Demo
src/gradio_demo.py - Inference CLI
inference.py - WebUI Extension
scripts/extension.py
Input Conditioning
- Portrait Prep
src/utils/preprocess.py - 3D Face Reconstruction
src/face3d/models/networks.py - Audio Features
src/generate_batch.py
Motion Models
- Audio Expression Model
src/audio2exp_models/audio2exp.py - Audio Pose Model
src/audio2pose_models/audio2pose.py
Face Rendering
- Render Data Prep
src/generate_facerender_batch.py - Animation Orchestrator
src/facerender/animate.py - Keypoint Detector
src/facerender/modules/keypoint_detector.py - Frame Generator
src/facerender/modules/generator.py - Face Enhancer
src/utils/face_enhancer.py
Other components
- User
- Talking Video
Connections 15
- User → Gradio Demo: submits inputs
README.md - User → Inference CLI: provides inputs
inference.py - Gradio Demo → Portrait Prep: prepares portrait
src/gradio_demo.py - Inference CLI → Portrait Prep: prepares portrait
inference.py - Portrait Prep → 3D Face Reconstruction: runs reconstruction
src/utils/preprocess.py - Gradio Demo → Audio Features: builds audio batch
src/gradio_demo.py - Audio Features → Audio Expression Model: conditions expression
- Audio Features → Audio Pose Model: conditions pose
- Gradio Demo → Render Data Prep: builds render data
src/gradio_demo.py - Render Data Prep → Animation Orchestrator: supplies render data
src/gradio_demo.py - Animation Orchestrator → Keypoint Detector: constructs detector
src/facerender/animate.py - Animation Orchestrator → Frame Generator: constructs generator
src/facerender/animate.py - Animation Orchestrator → Face Enhancer: optionally enhances
src/facerender/animate.py - Animation Orchestrator → Talking Video: writes video
src/facerender/animate.py - User → WebUI Extension: uses extension
README.md