How jdh-algo/joyvasa works
JoyVASA animates a human or animal reference portrait from speech. The main workflow accepts inputs through the CLI or Gradio, crops and encodes the reference, generates audio-conditioned motion, then renders and saves an animation. The README also describes the decoupled facial representation, animal support, and a training/data-preparation workflow. Relationships for the inference and UI paths are grounded in excerpts; some model internals and data-preparation connections are documented only at a high level or remain unsampled.
Main entry points: live_portrait_wmg_pipeline.py Human pipeline · live_portrait_wmg_pipeline_animal.py Animal pipeline · live_portrait_wmg_wrapper.py Portrait model wrapper · inference.py CLI inference
Components 25
User interfaces
- CLI inference
inference.py - Gradio app
app.py - Gradio pipeline
src/gradio_pipeline.py - Input arguments
src/config/argument_config.py
Animation inference
- Inference settings
src/config/inference_config.py - Crop settings
src/config/crop_config.py - Human pipeline
src/live_portrait_wmg_pipeline.py - Animal pipeline
src/live_portrait_wmg_pipeline_animal.py - Face cropper
src/utils/cropper.py - Crop transforms
src/utils/crop.py - Portrait model wrapper
src/live_portrait_wmg_wrapper.py - Camera transforms
src/utils/camera.py - Image and motion I/O
src/utils/io.py - Video output
src/utils/video.py
Motion and rendering models
- Diffusion motion model
src/modules/dit_talking_head.py - Audio encoders
src/modules/wav2vec2.py - Appearance encoder
src/modules/appearance_feature_extractor.py - Keypoint encoder
src/modules/motion_extractor.py - Warping network
src/modules/warping_network.py - Image generator
src/modules/spade_generator.py
Training and data
- Talking-head dataset
src/dataset/talkinghead_dataset_hungry.py - Model training
train.py - Motion and audio preparation
src/prepare_data - Motion template
src/prepare_data/04_gen_template.py
Other components
- User
Connections 31
- User → CLI inference: provides inputs
inference.py - User → Gradio app: uploads media
README.md - Gradio app → Gradio pipeline: starts interface
app.py - CLI inference → Input arguments: parses options
inference.py - CLI inference → Human pipeline: runs human mode
inference.py - CLI inference → Animal pipeline: runs animal mode
inference.py - Gradio pipeline → Human pipeline: executes
src/gradio_pipeline.py - Gradio pipeline → Animal pipeline: executes
src/gradio_pipeline.py - Human pipeline → Face cropper: crops reference
src/live_portrait_wmg_pipeline.py - Animal pipeline → Face cropper: crops reference
src/live_portrait_wmg_pipeline_animal.py - Face cropper → Crop transforms: uses transforms
src/utils/cropper.py - Human pipeline → Portrait model wrapper: extracts and renders
src/live_portrait_wmg_pipeline.py - Animal pipeline → Portrait model wrapper: extracts and renders
src/live_portrait_wmg_pipeline_animal.py - Human pipeline → Diffusion motion model: samples motion
README.md - Animal pipeline → Diffusion motion model: samples motion
README.md - Diffusion motion model → Audio encoders: encodes speech
src/modules/dit_talking_head.py - Human pipeline → Camera transforms: transforms keypoints
src/live_portrait_wmg_pipeline.py - Animal pipeline → Camera transforms: transforms keypoints
src/live_portrait_wmg_pipeline_animal.py - Human pipeline → Image and motion I/O: loads media
src/live_portrait_wmg_pipeline.py - Animal pipeline → Image and motion I/O: loads media
src/live_portrait_wmg_pipeline_animal.py - Human pipeline → Video output: writes animation
src/live_portrait_wmg_pipeline.py - Animal pipeline → Video output: writes animation
src/live_portrait_wmg_pipeline_animal.py - Portrait model wrapper → Appearance encoder: loads encoder
- Portrait model wrapper → Keypoint encoder: loads encoder
- Portrait model wrapper → Warping network: loads network
- Portrait model wrapper → Image generator: loads generator
- Motion and audio preparation → Motion template: generates template
- Model training → Talking-head dataset: uses dataset
train.py - Model training → Diffusion motion model: trains model
train.py - User → Human pipeline: selects portrait mode
README.md - User → Animal pipeline: selects animal mode
README.md