# jdh-algo/joyvasa architecture

> Architecture diagram of the GitHub repository jdh-algo/joyvasa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jdh-algo/joyvasa
- Repository: https://github.com/jdh-algo/joyvasa

## Overview

JoyVASA animates a human or animal reference portrait from speech. The main workflow accepts inputs through the CLI or Gradio, crops and encodes the reference, generates audio-conditioned motion, then renders and saves an animation. The README also describes the decoupled facial representation, animal support, and a training/data-preparation workflow. Relationships for the inference and UI paths are grounded in excerpts; some model internals and data-preparation connections are documented only at a high level or remain unsampled.

Main entry points: [`src/live_portrait_wmg_pipeline.py`](https://github.com/jdh-algo/joyvasa/blob/main/src/live_portrait_wmg_pipeline.py) (Human pipeline), [`src/live_portrait_wmg_pipeline_animal.py`](https://github.com/jdh-algo/joyvasa/blob/main/src/live_portrait_wmg_pipeline_animal.py) (Animal pipeline), [`src/live_portrait_wmg_wrapper.py`](https://github.com/jdh-algo/joyvasa/blob/main/src/live_portrait_wmg_wrapper.py) (Portrait model wrapper), [`inference.py`](https://github.com/jdh-algo/joyvasa/blob/main/inference.py) (CLI inference)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["User interfaces"]
  node_cli["CLI inference<br/>[inference.py]"]
  node_web["Gradio app<br/>[app.py]"]
  node_gradio["Gradio pipeline<br/>[gradio_pipeline.py]"]
  node_args["Input arguments<br/>[argument_config.py]"]
end

subgraph group_inference["Animation inference"]
  node_infercfg["Inference settings"]
  node_cropcfg["Crop settings<br/>[crop_config.py]"]
  node_human["Human pipeline"]
  node_animal["Animal pipeline"]
  node_cropper["Face cropper<br/>[cropper.py]"]
  node_crop["Crop transforms<br/>[crop.py]"]
  node_wrapper["Portrait model wrapper"]
  node_camera["Camera transforms<br/>[camera.py]"]
  node_io["Image and motion I/O<br/>[io.py]"]
  node_video["Video output<br/>[video.py]"]
end

subgraph group_models["Motion and rendering models"]
  node_motion["Diffusion motion model"]
  node_audio["Audio encoders<br/>[wav2vec2.py]"]
  node_appearance["Appearance encoder"]
  node_motionenc["Keypoint encoder"]
  node_warping["Warping network<br/>[warping_network.py]"]
  node_generator["Image generator<br/>[spade_generator.py]"]
end

subgraph group_data["Training and data"]
  node_dataset["Talking-head dataset"]
  node_train["Model training<br/>[train.py]"]
  node_prep["Motion and audio preparation"]
  node_template["Motion template<br/>[04_gen_template.py]"]
end

node_user(("User"))

node_user -->|"provides inputs"| node_cli
node_user -->|"uploads media"| node_web
node_web -->|"starts interface"| node_gradio
node_cli -->|"parses options"| node_args
node_cli -->|"runs human mode"| node_human
node_cli -->|"runs animal mode"| node_animal
node_gradio -->|"executes"| node_human
node_gradio -->|"executes"| node_animal
node_human -->|"crops reference"| node_cropper
node_animal -->|"crops reference"| node_cropper
node_cropper -->|"uses transforms"| node_crop
node_human -->|"extracts and renders"| node_wrapper
node_animal -->|"extracts and renders"| node_wrapper
node_human -.->|"samples motion"| node_motion
node_animal -.->|"samples motion"| node_motion
node_motion -->|"encodes speech"| node_audio
node_human -->|"transforms keypoints"| node_camera
node_animal -->|"transforms keypoints"| node_camera
node_human -->|"loads media"| node_io
node_animal -->|"loads media"| node_io
node_human -->|"writes animation"| node_video
node_animal -->|"writes animation"| node_video
node_wrapper -->|"loads encoder"| node_appearance
node_wrapper -->|"loads encoder"| node_motionenc
node_wrapper -->|"loads network"| node_warping
node_wrapper -->|"loads generator"| node_generator
node_prep -.->|"generates template"| node_template
node_train -.->|"uses dataset"| node_dataset
node_train -->|"trains model"| node_motion
node_user -.->|"selects portrait mode"| node_human
node_user -.->|"selects animal mode"| node_animal

click node_cli "https://github.com/jdh-algo/joyvasa/blob/main/inference.py"
click node_web "https://github.com/jdh-algo/joyvasa/blob/main/app.py"
click node_gradio "https://github.com/jdh-algo/joyvasa/blob/main/src/gradio_pipeline.py"
click node_args "https://github.com/jdh-algo/joyvasa/blob/main/src/config/argument_config.py"
click node_infercfg "https://github.com/jdh-algo/joyvasa/blob/main/src/config/inference_config.py"
click node_cropcfg "https://github.com/jdh-algo/joyvasa/blob/main/src/config/crop_config.py"
click node_human "https://github.com/jdh-algo/joyvasa/blob/main/src/live_portrait_wmg_pipeline.py"
click node_animal "https://github.com/jdh-algo/joyvasa/blob/main/src/live_portrait_wmg_pipeline_animal.py"
click node_cropper "https://github.com/jdh-algo/joyvasa/blob/main/src/utils/cropper.py"
click node_crop "https://github.com/jdh-algo/joyvasa/blob/main/src/utils/crop.py"
click node_wrapper "https://github.com/jdh-algo/joyvasa/blob/main/src/live_portrait_wmg_wrapper.py"
click node_camera "https://github.com/jdh-algo/joyvasa/blob/main/src/utils/camera.py"
click node_io "https://github.com/jdh-algo/joyvasa/blob/main/src/utils/io.py"
click node_video "https://github.com/jdh-algo/joyvasa/blob/main/src/utils/video.py"
click node_motion "https://github.com/jdh-algo/joyvasa/blob/main/src/modules/dit_talking_head.py"
click node_audio "https://github.com/jdh-algo/joyvasa/blob/main/src/modules/wav2vec2.py"
click node_appearance "https://github.com/jdh-algo/joyvasa/blob/main/src/modules/appearance_feature_extractor.py"
click node_motionenc "https://github.com/jdh-algo/joyvasa/blob/main/src/modules/motion_extractor.py"
click node_warping "https://github.com/jdh-algo/joyvasa/blob/main/src/modules/warping_network.py"
click node_generator "https://github.com/jdh-algo/joyvasa/blob/main/src/modules/spade_generator.py"
click node_dataset "https://github.com/jdh-algo/joyvasa/blob/main/src/dataset/talkinghead_dataset_hungry.py"
click node_train "https://github.com/jdh-algo/joyvasa/blob/main/train.py"
click node_prep "https://github.com/jdh-algo/joyvasa/tree/main/src/prepare_data"
click node_template "https://github.com/jdh-algo/joyvasa/blob/main/src/prepare_data/04_gen_template.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_web,node_gradio,node_args,node_user toneBlue
class node_infercfg,node_cropcfg,node_human,node_animal,node_cropper,node_crop,node_wrapper,node_camera,node_io,node_video toneAmber
class node_motion,node_audio,node_appearance,node_motionenc,node_warping,node_generator toneMint
class node_dataset,node_train,node_prep,node_template toneRose
```

## Components

### User interfaces

- [CLI inference](https://github.com/jdh-algo/joyvasa/blob/main/inference.py): `inference.py`
- [Gradio app](https://github.com/jdh-algo/joyvasa/blob/main/app.py): `app.py`
- [Gradio pipeline](https://github.com/jdh-algo/joyvasa/blob/main/src/gradio_pipeline.py): `src/gradio_pipeline.py`
- [Input arguments](https://github.com/jdh-algo/joyvasa/blob/main/src/config/argument_config.py): `src/config/argument_config.py`

### Animation inference

- [Inference settings](https://github.com/jdh-algo/joyvasa/blob/main/src/config/inference_config.py): `src/config/inference_config.py`
- [Crop settings](https://github.com/jdh-algo/joyvasa/blob/main/src/config/crop_config.py): `src/config/crop_config.py`
- [Human pipeline](https://github.com/jdh-algo/joyvasa/blob/main/src/live_portrait_wmg_pipeline.py): `src/live_portrait_wmg_pipeline.py`
- [Animal pipeline](https://github.com/jdh-algo/joyvasa/blob/main/src/live_portrait_wmg_pipeline_animal.py): `src/live_portrait_wmg_pipeline_animal.py`
- [Face cropper](https://github.com/jdh-algo/joyvasa/blob/main/src/utils/cropper.py): `src/utils/cropper.py`
- [Crop transforms](https://github.com/jdh-algo/joyvasa/blob/main/src/utils/crop.py): `src/utils/crop.py`
- [Portrait model wrapper](https://github.com/jdh-algo/joyvasa/blob/main/src/live_portrait_wmg_wrapper.py): `src/live_portrait_wmg_wrapper.py`
- [Camera transforms](https://github.com/jdh-algo/joyvasa/blob/main/src/utils/camera.py): `src/utils/camera.py`
- [Image and motion I/O](https://github.com/jdh-algo/joyvasa/blob/main/src/utils/io.py): `src/utils/io.py`
- [Video output](https://github.com/jdh-algo/joyvasa/blob/main/src/utils/video.py): `src/utils/video.py`

### Motion and rendering models

- [Diffusion motion model](https://github.com/jdh-algo/joyvasa/blob/main/src/modules/dit_talking_head.py): `src/modules/dit_talking_head.py`
- [Audio encoders](https://github.com/jdh-algo/joyvasa/blob/main/src/modules/wav2vec2.py): `src/modules/wav2vec2.py`
- [Appearance encoder](https://github.com/jdh-algo/joyvasa/blob/main/src/modules/appearance_feature_extractor.py): `src/modules/appearance_feature_extractor.py`
- [Keypoint encoder](https://github.com/jdh-algo/joyvasa/blob/main/src/modules/motion_extractor.py): `src/modules/motion_extractor.py`
- [Warping network](https://github.com/jdh-algo/joyvasa/blob/main/src/modules/warping_network.py): `src/modules/warping_network.py`
- [Image generator](https://github.com/jdh-algo/joyvasa/blob/main/src/modules/spade_generator.py): `src/modules/spade_generator.py`

### Training and data

- [Talking-head dataset](https://github.com/jdh-algo/joyvasa/blob/main/src/dataset/talkinghead_dataset_hungry.py): `src/dataset/talkinghead_dataset_hungry.py`
- [Model training](https://github.com/jdh-algo/joyvasa/blob/main/train.py): `train.py`
- [Motion and audio preparation](https://github.com/jdh-algo/joyvasa/tree/main/src/prepare_data): `src/prepare_data`
- [Motion template](https://github.com/jdh-algo/joyvasa/blob/main/src/prepare_data/04_gen_template.py): `src/prepare_data/04_gen_template.py`

### Other components

- **User**

## Connections

- User → CLI inference: provides inputs (evidence: `inference.py`)
- User → Gradio app: uploads media (evidence: `README.md`)
- Gradio app → Gradio pipeline: starts interface (evidence: `app.py`)
- CLI inference → Input arguments: parses options (evidence: `inference.py`)
- CLI inference → Human pipeline: runs human mode (evidence: `inference.py`)
- CLI inference → Animal pipeline: runs animal mode (evidence: `inference.py`)
- Gradio pipeline → Human pipeline: executes (evidence: `src/gradio_pipeline.py`)
- Gradio pipeline → Animal pipeline: executes (evidence: `src/gradio_pipeline.py`)
- Human pipeline → Face cropper: crops reference (evidence: `src/live_portrait_wmg_pipeline.py`)
- Animal pipeline → Face cropper: crops reference (evidence: `src/live_portrait_wmg_pipeline_animal.py`)
- Face cropper → Crop transforms: uses transforms (evidence: `src/utils/cropper.py`)
- Human pipeline → Portrait model wrapper: extracts and renders (evidence: `src/live_portrait_wmg_pipeline.py`)
- Animal pipeline → Portrait model wrapper: extracts and renders (evidence: `src/live_portrait_wmg_pipeline_animal.py`)
- Human pipeline → Diffusion motion model: samples motion (evidence: `README.md`)
- Animal pipeline → Diffusion motion model: samples motion (evidence: `README.md`)
- Diffusion motion model → Audio encoders: encodes speech (evidence: `src/modules/dit_talking_head.py`)
- Human pipeline → Camera transforms: transforms keypoints (evidence: `src/live_portrait_wmg_pipeline.py`)
- Animal pipeline → Camera transforms: transforms keypoints (evidence: `src/live_portrait_wmg_pipeline_animal.py`)
- Human pipeline → Image and motion I/O: loads media (evidence: `src/live_portrait_wmg_pipeline.py`)
- Animal pipeline → Image and motion I/O: loads media (evidence: `src/live_portrait_wmg_pipeline_animal.py`)
- Human pipeline → Video output: writes animation (evidence: `src/live_portrait_wmg_pipeline.py`)
- Animal pipeline → Video output: writes animation (evidence: `src/live_portrait_wmg_pipeline_animal.py`)
- Portrait model wrapper → Appearance encoder: loads encoder
- Portrait model wrapper → Keypoint encoder: loads encoder
- Portrait model wrapper → Warping network: loads network
- Portrait model wrapper → Image generator: loads generator
- Motion and audio preparation → Motion template: generates template
- Model training → Talking-head dataset: uses dataset (evidence: `train.py`)
- Model training → Diffusion motion model: trains model (evidence: `train.py`)
- User → Human pipeline: selects portrait mode (evidence: `README.md`)
- User → Animal pipeline: selects animal mode (evidence: `README.md`)
