How alibaba-quark/liveavatar works
LiveAvatar is a streaming audio-driven avatar-video generation system, exposed through a Gradio interface and command-line inference. The documented workflow accepts audio with a reference image and optional prompt, then uses a causal speech-to-video pipeline for streaming, block-wise autoregressive generation; distributed inference and streaming VAE support are key internal layers. The repository also includes Wan text/video and image/video capabilities. Connections below follow the README and source-index references; unsampled implementation details are represented by their exact tree paths, without inferring unshown calls.
Main entry points: causal_s2v_pipeline.py Causal S2V · causal_s2v_pipeline_tpp.py TPP streaming · s2v_streaming_interact.py Streaming CLI · batch_eval.py Batch inference
Components 21
Inference interfaces
- Gradio UI
minimal_inference/gradio_app.py - Streaming CLI
minimal_inference/s2v_streaming_interact.py - Batch inference
minimal_inference/batch_eval.py
Generation pipelines
- Causal S2V
liveavatar/models/wan/causal_s2v_pipeline.py - TPP streaming
liveavatar/models/wan/causal_s2v_pipeline_tpp.py - Blockwise TPP
liveavatar/models/wan/causal_s2v_pipeline_tpp_blockwise.py - Speech-video pipeline
liveavatar/models/wan/wan_2_2/speech2video.py
Model components
- Causal S2V model
liveavatar/models/wan/causal_model_s2v.py - S2V model blocks
liveavatar/models/wan/wan_2_2/modules/s2v/model_s2v.py - Text encoder
liveavatar/models/wan/wan_2_2/modules/t5.py - Video VAE
liveavatar/models/wan/wan_2_2/modules/vae2_1.py - Streaming VAE
liveavatar/models/wan/wan_2_2/modules/vae_streaming.py - Flow samplers
liveavatar/models/wan/wan_2_2/utils/fm_solvers.py - Flow matching
liveavatar/models/wan/flow_match.py - Wan base model
liveavatar/models/wan/wan_base/modules/model.py
Distributed runtime
- Sequence parallel
liveavatar/models/wan/wan_2_2/distributed/sequence_parallel.py - FSDP sharding
liveavatar/models/wan/wan_2_2/distributed/fsdp.py
Prompt and checkpoint support
- Inference options
liveavatar/utils/args_config.py - Weight loading
liveavatar/utils/load_weight_utils.py - Prompt extension
liveavatar/models/wan/wan_2_2/utils/prompt_extend.py
Other components
- User
Connections 30
- User → Gradio UI: submits request
- User → Streaming CLI: provides inputs
- User → Batch inference: provides inputs
- Gradio UI → TPP streaming: uses pipeline
minimal_inference/gradio_app.py - Gradio UI → Causal S2V: uses pipeline
minimal_inference/gradio_app.py - Streaming CLI → TPP streaming: uses pipeline
minimal_inference/s2v_streaming_interact.py - Streaming CLI → Causal S2V: uses pipeline
minimal_inference/s2v_streaming_interact.py - Batch inference → TPP streaming: uses pipeline
minimal_inference/batch_eval.py - Batch inference → Causal S2V: uses pipeline
minimal_inference/batch_eval.py - Gradio UI → Inference options: loads options
minimal_inference/gradio_app.py - Streaming CLI → Inference options: loads options
minimal_inference/s2v_streaming_interact.py - Batch inference → Inference options: loads options
minimal_inference/batch_eval.py - Causal S2V → Causal S2V model: imports model
liveavatar/models/wan/causal_s2v_pipeline.py - Causal S2V → Text encoder: imports encoder
liveavatar/models/wan/causal_s2v_pipeline.py - Causal S2V → Video VAE: imports VAE
liveavatar/models/wan/causal_s2v_pipeline.py - Causal S2V → Flow samplers: imports solver
liveavatar/models/wan/causal_s2v_pipeline.py - Causal S2V → Flow matching: imports utilities
liveavatar/models/wan/causal_s2v_pipeline.py - Causal S2V → Weight loading: loads weights
liveavatar/models/wan/causal_s2v_pipeline.py - TPP streaming → Streaming VAE: imports VAE
liveavatar/models/wan/causal_s2v_pipeline_tpp.py - TPP streaming → Sequence parallel: imports parallel runtime
liveavatar/models/wan/causal_s2v_pipeline_tpp.py - TPP streaming → FSDP sharding: imports sharding
liveavatar/models/wan/causal_s2v_pipeline_tpp.py - TPP streaming → Causal S2V model: imports model
liveavatar/models/wan/causal_s2v_pipeline_tpp.py - TPP streaming → Flow samplers: imports solver
liveavatar/models/wan/causal_s2v_pipeline_tpp.py - Speech-video pipeline → S2V model blocks: imports model
liveavatar/models/wan/wan_2_2/speech2video.py - Speech-video pipeline → Text encoder: imports encoder
liveavatar/models/wan/wan_2_2/speech2video.py - Speech-video pipeline → Video VAE: imports VAE
liveavatar/models/wan/wan_2_2/speech2video.py - Causal S2V model → S2V model blocks: imports model blocks
liveavatar/models/wan/causal_model_s2v.py - Sequence parallel → Wan base model: imports embeddings
liveavatar/models/wan/wan_2_2/distributed/sequence_parallel.py - Batch inference → Prompt extension: imports extension
minimal_inference/batch_eval.py - Streaming CLI → Prompt extension: imports extension
minimal_inference/s2v_streaming_interact.py