How francis-rings/stableavatar works
StableAvatar generates identity-preserving, audio-driven avatar video from a reference image and speech audio, with long-duration output and training/data-preparation workflows. The main user path runs through the Gradio app into the inference pipeline, which conditions video diffusion on image, text and audio features, then decodes and muxes the result. The graph also captures documented preprocessing and training capabilities. Some source excerpts are partial, so unconfirmed internal calls and subsystem wiring are omitted.
Main entry points: wan_inference_long_pipeline.py Long-video pipeline · app.py Gradio interface · talking_video_dataset_fantasy.py Talking-video dataset · wan_fantasy_transformer3d_1B.py Video transformer
Components 23
Video generation
- Gradio interface
app.py - Inference runner
inference.py - Long-video pipeline
wan/pipeline/wan_inference_long_pipeline.py - Fantasy pipeline
wan/pipeline/wan_inference_pipeline_fantasy.py - Text-to-video runner
wan/text2video.py - Image-to-video runner
wan/image2video.py
Model components
- Video transformer
wan/models/wan_fantasy_transformer3d_1B.py - Vocal projector
wan/models/vocal_projector_fantasy_1B.py - Audio encoder
wan/models/wav2vec.py - Text encoder
wan/models/wan_text_encoder.py - Image encoder
wan/models/wan_image_encoder.py - Video VAE
wan/models/wan_vae.py - Motion controller
wan/models/motion_controller.py
Audio preparation
- Audio extraction
audio_extractor.py - Vocal separation
vocal_seperator.py - Lip-mask extraction
lip_mask_extractor.py
Training
- Talking-video dataset
wan/dataset/talking_video_dataset_fantasy.py - Training and LoRA
train_1B_square.py
Other components
- User
- Generated video
- Speech audio
- Reference image
- Prompt
Connections 22
- User → Gradio interface: provides inputs
- Gradio interface → Long-video pipeline: runs generation
- Inference runner → Long-video pipeline: runs generation
- Long-video pipeline → Text encoder: encodes prompt
- Long-video pipeline → Image encoder: encodes image
- Long-video pipeline → Audio encoder: encodes audio
- Long-video pipeline → Video transformer: denoises latents
- Long-video pipeline → Vocal projector: projects audio
- Long-video pipeline → Video VAE: decodes frames
- Long-video pipeline → Generated video: produces
- Gradio interface → Speech audio: reads audio
- Gradio interface → Reference image: reads image
- Gradio interface → Prompt: reads prompt
- Gradio interface → Generated video: muxes audio
- Text-to-video runner → Fantasy pipeline: uses pipeline
- Image-to-video runner → Video transformer: runs model
- Talking-video dataset → Training and LoRA: supplies samples
- Training and LoRA → Video transformer: trains model
- Audio extraction → Speech audio: extracts tracks
- Vocal separation → Speech audio: separates vocals
- Lip-mask extraction → Talking-video dataset: prepares masks
- Talking-video dataset → Speech audio: loads audio