How tencentarc/tooncomposer works
ToonComposer is a generative cartoon post-keyframing application: users provide a prompt, color keyframes, sketches and optional motion masks, then generate a video. The graph follows the documented UI workflow into conditioning and model inference, with separate model, sampling, and checkpoint-loading layers. README documents controls and two weight sources; excerpts confirm app initialization and key model components, but not every UI callback or the full denoising call chain, so uncertain wiring is omitted.
Main entry points: i2v_pipeline.py Video inference pipeline · app.py Gradio app · tooncomposer.py Model initialization · prompter.py Prompt tokenizer
Components 21
User interface
- Gradio app
app.py - Text prompt
- Color keyframes
- Sketches and masks
Conditioning and generation
- Condition processing
app.py - ToonComposer
tooncomposer.py - Video inference pipeline
pipeline/i2v_pipeline.py
Model components
- Text encoder
model/text_encoder.py - Prompt tokenizer
model/prompter.py - Image encoder
model/image_encoder.py - Denoising transformer
model/dit.py - DeRA attention
model/dera.py - Video VAE
model/vae.py
Loading and sampling
- Flow scheduler
scheduler/flow_match.py - Model initialization
tooncomposer.py - Weight resolution
app.py
Other components
- Artist
- Generated video
- Hugging Face Hub
- Wan2.1 weights
- ToonComposer weights
Connections 24
- Artist → Gradio app: uses
- Gradio app → Condition processing: processes inputs
- Text prompt → Gradio app: sets prompt
- Color keyframes → Condition processing: conditions frames
- Sketches and masks → Condition processing: conditions motion
- Condition processing → ToonComposer: passes conditions
- Gradio app → ToonComposer: invokes generation
- ToonComposer → Video inference pipeline: uses pipeline
- Video inference pipeline → Generated video: returns video
- Video inference pipeline → Prompt tokenizer: tokenizes prompt
- Prompt tokenizer → Text encoder: encodes text
- Video inference pipeline → Image encoder: encodes images
- Video inference pipeline → Denoising transformer: denoises latents
- ToonComposer → DeRA attention: configures attention
- Video inference pipeline → Flow scheduler: updates latents
- Video inference pipeline → Video VAE: decodes video
- Gradio app → Weight resolution: resolves weights
- Weight resolution → Hugging Face Hub: checks cache or downloads
- Weight resolution → Wan2.1 weights: locates foundation weights
- Weight resolution → ToonComposer weights: locates composer weights
- Gradio app → Model initialization: initializes model
- Model initialization → Video inference pipeline: loads components
- Model initialization → Wan2.1 weights: loads foundation
- Model initialization → ToonComposer weights: loads fine-tune