# tencentarc/tooncomposer architecture

> Architecture diagram of the GitHub repository tencentarc/tooncomposer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencentarc/tooncomposer
- Repository: https://github.com/tencentarc/tooncomposer

## Overview

ToonComposer is a generative cartoon post-keyframing application: users provide a prompt, color keyframes, sketches and optional motion masks, then generate a video. The graph follows the documented UI workflow into conditioning and model inference, with separate model, sampling, and checkpoint-loading layers. README documents controls and two weight sources; excerpts confirm app initialization and key model components, but not every UI callback or the full denoising call chain, so uncertain wiring is omitted.

Main entry points: [`pipeline/i2v_pipeline.py`](https://github.com/tencentarc/tooncomposer/blob/main/pipeline/i2v_pipeline.py) (Video inference pipeline), [`app.py`](https://github.com/tencentarc/tooncomposer/blob/main/app.py) (Gradio app), [`tooncomposer.py`](https://github.com/tencentarc/tooncomposer/blob/main/tooncomposer.py) (Model initialization), [`model/prompter.py`](https://github.com/tencentarc/tooncomposer/blob/main/model/prompter.py) (Prompt tokenizer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User interface"]
  node_app["Gradio app<br/>[app.py]"]
  node_prompt["Text prompt"]
  node_keyframes["Color keyframes"]
  node_sketches["Sketches and masks"]
end

subgraph group_generation["Conditioning and generation"]
  node_condition_processing["Condition processing<br/>[app.py]"]
  node_composer["ToonComposer<br/>[tooncomposer.py]"]
  node_inference_pipeline["Video inference pipeline<br/>[i2v_pipeline.py]"]
end

subgraph group_models["Model components"]
  node_text_encoder["Text encoder<br/>[text_encoder.py]"]
  node_prompter["Prompt tokenizer<br/>[prompter.py]"]
  node_image_encoder["Image encoder<br/>[image_encoder.py]"]
  node_dit["Denoising transformer<br/>[dit.py]"]
  node_dera["DeRA attention<br/>[dera.py]"]
  node_vae["Video VAE<br/>[vae.py]"]
end

subgraph group_runtime["Loading and sampling"]
  node_scheduler["Flow scheduler<br/>[flow_match.py]"]
  node_model_loading["Model initialization<br/>[tooncomposer.py]"]
  node_weight_resolution["Weight resolution<br/>[app.py]"]
end

node_artist(("Artist"))
node_video["Generated video"]
node_huggingface["Hugging Face Hub"]
node_wan_weights["Wan2.1 weights"]
node_toon_weights["ToonComposer weights"]

node_artist -->|"uses"| node_app
node_app -->|"processes inputs"| node_condition_processing
node_prompt -->|"sets prompt"| node_app
node_keyframes -->|"conditions frames"| node_condition_processing
node_sketches -->|"conditions motion"| node_condition_processing
node_condition_processing -->|"passes conditions"| node_composer
node_app -->|"invokes generation"| node_composer
node_composer -->|"uses pipeline"| node_inference_pipeline
node_inference_pipeline -->|"returns video"| node_video
node_inference_pipeline -->|"tokenizes prompt"| node_prompter
node_prompter -->|"encodes text"| node_text_encoder
node_inference_pipeline -->|"encodes images"| node_image_encoder
node_inference_pipeline -->|"denoises latents"| node_dit
node_composer -->|"configures attention"| node_dera
node_inference_pipeline -->|"updates latents"| node_scheduler
node_inference_pipeline -->|"decodes video"| node_vae
node_app -->|"resolves weights"| node_weight_resolution
node_weight_resolution -.->|"checks cache or downloads"| node_huggingface
node_weight_resolution -->|"locates foundation weights"| node_wan_weights
node_weight_resolution -->|"locates composer weights"| node_toon_weights
node_app -->|"initializes model"| node_model_loading
node_model_loading -->|"loads components"| node_inference_pipeline
node_model_loading -->|"loads foundation"| node_wan_weights
node_model_loading -->|"loads fine-tune"| node_toon_weights

click node_app "https://github.com/tencentarc/tooncomposer/blob/main/app.py"
click node_condition_processing "https://github.com/tencentarc/tooncomposer/blob/main/app.py"
click node_composer "https://github.com/tencentarc/tooncomposer/blob/main/tooncomposer.py"
click node_inference_pipeline "https://github.com/tencentarc/tooncomposer/blob/main/pipeline/i2v_pipeline.py"
click node_text_encoder "https://github.com/tencentarc/tooncomposer/blob/main/model/text_encoder.py"
click node_prompter "https://github.com/tencentarc/tooncomposer/blob/main/model/prompter.py"
click node_image_encoder "https://github.com/tencentarc/tooncomposer/blob/main/model/image_encoder.py"
click node_dit "https://github.com/tencentarc/tooncomposer/blob/main/model/dit.py"
click node_dera "https://github.com/tencentarc/tooncomposer/blob/main/model/dera.py"
click node_vae "https://github.com/tencentarc/tooncomposer/blob/main/model/vae.py"
click node_scheduler "https://github.com/tencentarc/tooncomposer/blob/main/scheduler/flow_match.py"
click node_model_loading "https://github.com/tencentarc/tooncomposer/blob/main/tooncomposer.py"
click node_weight_resolution "https://github.com/tencentarc/tooncomposer/blob/main/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_prompt,node_keyframes,node_sketches toneBlue
class node_condition_processing,node_composer,node_inference_pipeline toneAmber
class node_text_encoder,node_prompter,node_image_encoder,node_dit,node_dera,node_vae toneMint
class node_scheduler,node_model_loading,node_weight_resolution toneRose
class node_artist,node_video,node_huggingface,node_wan_weights,node_toon_weights toneIndigo
```

## Components

### User interface

- [Gradio app](https://github.com/tencentarc/tooncomposer/blob/main/app.py): `app.py`
- **Text prompt**
- **Color keyframes**
- **Sketches and masks**

### Conditioning and generation

- [Condition processing](https://github.com/tencentarc/tooncomposer/blob/main/app.py): `app.py`
- [ToonComposer](https://github.com/tencentarc/tooncomposer/blob/main/tooncomposer.py): `tooncomposer.py`
- [Video inference pipeline](https://github.com/tencentarc/tooncomposer/blob/main/pipeline/i2v_pipeline.py): `pipeline/i2v_pipeline.py`

### Model components

- [Text encoder](https://github.com/tencentarc/tooncomposer/blob/main/model/text_encoder.py): `model/text_encoder.py`
- [Prompt tokenizer](https://github.com/tencentarc/tooncomposer/blob/main/model/prompter.py): `model/prompter.py`
- [Image encoder](https://github.com/tencentarc/tooncomposer/blob/main/model/image_encoder.py): `model/image_encoder.py`
- [Denoising transformer](https://github.com/tencentarc/tooncomposer/blob/main/model/dit.py): `model/dit.py`
- [DeRA attention](https://github.com/tencentarc/tooncomposer/blob/main/model/dera.py): `model/dera.py`
- [Video VAE](https://github.com/tencentarc/tooncomposer/blob/main/model/vae.py): `model/vae.py`

### Loading and sampling

- [Flow scheduler](https://github.com/tencentarc/tooncomposer/blob/main/scheduler/flow_match.py): `scheduler/flow_match.py`
- [Model initialization](https://github.com/tencentarc/tooncomposer/blob/main/tooncomposer.py): `tooncomposer.py`
- [Weight resolution](https://github.com/tencentarc/tooncomposer/blob/main/app.py): `app.py`

### Other components

- **Artist**
- **Generated video**
- **Hugging Face Hub**
- **Wan2.1 weights**
- **ToonComposer weights**

## Connections

- Artist → Gradio app: uses
- Gradio app → Condition processing: processes inputs
- Text prompt → Gradio app: sets prompt
- Color keyframes → Condition processing: conditions frames
- Sketches and masks → Condition processing: conditions motion
- Condition processing → ToonComposer: passes conditions
- Gradio app → ToonComposer: invokes generation
- ToonComposer → Video inference pipeline: uses pipeline
- Video inference pipeline → Generated video: returns video
- Video inference pipeline → Prompt tokenizer: tokenizes prompt
- Prompt tokenizer → Text encoder: encodes text
- Video inference pipeline → Image encoder: encodes images
- Video inference pipeline → Denoising transformer: denoises latents
- ToonComposer → DeRA attention: configures attention
- Video inference pipeline → Flow scheduler: updates latents
- Video inference pipeline → Video VAE: decodes video
- Gradio app → Weight resolution: resolves weights
- Weight resolution → Hugging Face Hub: checks cache or downloads
- Weight resolution → Wan2.1 weights: locates foundation weights
- Weight resolution → ToonComposer weights: locates composer weights
- Gradio app → Model initialization: initializes model
- Model initialization → Video inference pipeline: loads components
- Model initialization → Wan2.1 weights: loads foundation
- Model initialization → ToonComposer weights: loads fine-tune