# doubiiu/tooncrafter architecture

> Architecture diagram of the GitHub repository doubiiu/tooncrafter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/doubiiu/tooncrafter
- Repository: https://github.com/doubiiu/tooncrafter

## Overview

##### Project type and purpose
- ToonCrafter is an open-source research tool for generative video interpolation of cartoon frames. It provides:
- Training pipeline (configs, main/trainer, lvdm modules)
- Inference pipeline (CLI scripts, Gradio app)
- Core diffusion model library (lvdm/)
- Utilities for data loading, evaluation, saving videos

##### File structure
- Top-level directories:
- configs/ – YAML configs and run.sh for different resolutions
- lvdm/ – core diffusion library: data loaders, model definitions, samplers, modules
- main/ – training entrypoint, callbacks, utilities
- scripts/ – evaluation, inference, Gradio demos, run.sh
- utils/, prompts/, assets/
- gradio_app.py – standalone UI
- Checkpoints and external model weights live under checkpoints/ (created at inference)
- requirements.txt, LICENSE, README.md

##### README for additional insights
- Describes use cases: cartoon interpolation, sketch guidance, sketch colorization
- Details on model resolution, GPU memory, links to Hugging Face checkpoints
- Instructions for CLI inference (scripts/run.sh) and Gradio demo

a. Main components
1. User Interfaces
– CLI wrapper (scripts/run.sh → scripts/evaluation/inference.py)
– Local Gradio UI (gradio_app.py + scripts/gradio/)
2. Configuration
– configs/*.yaml & run.sh
3. Core Library (lvdm/)
– Data loaders (lvdm/data/)
– Model definitions (lvdm/models/, samplers)
– Modules & networks (lvdm/modules/)
– Diffusion utilities (lvdm/basics.py, distributions.py, ema.py)
4. Training Pipeline (main/)
– Trainer (main/trainer.py)
– Callbacks (main/callbacks.py)
– Data & training utilities (main/utils_*.py)
5. Inference Pipeline
– Scripts (scripts/evaluation/inference.py)
– Video saving (utils/save_video.py)
6. External Dependencies
– Pretrained checkpoints (Hugging Face)
– Python libs (requirements.txt)

b. Relationships & interactions
- User → CLI or UI → loads config → loads model checkpoint → calls lvdm core to run diffusion inference → outputs video → utils/save_video
- Training: main/trainer reads configs → data loaders → lvdm model → training loop → callbacks → save checkpoint
- Scripts import lvdm modules and utils

c. Architectural patterns & design principles
- Modular separation: core library vs. application scripts
- Config-driven design: all hyperparameters in configs/
- Clear split between training and inference code paths
- Extensibility: additional samplers, modules, guidance easily plugged in under lvdm/

d. Key technologies & libraries
- PyTorch (models, training loop)
- YAML configs (Hydra-style)
- Gradio (UI)
- FFmpeg or OpenCV (video saving)
- Shell scripts for orchestration

##### Detailed breakdown of components
- Config manager
- Data loader
- Model encoder/decoder modules
- Sampler (DDIM, multi-cond)
- EMA & diffusion utilities
- Training orchestrator (trainer + callbacks)
- Inference engine (script + video saver)
- CLI & Gradio frontends
- Checkpoint store & assets

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interfaces
    subgraph "User Interfaces"
        direction TB
        CLI["CLI<br/>scripts/run.sh"]:::frontend
        Gradio["Gradio UI<br/>gradio_app.py"]:::frontend
    end

    %% Configuration
    Config["Config Manager<br/>configs/"]:::config

    %% Core Library
    subgraph "Core Library (lvdm/)" 
        direction TB
        subgraph "Data Loaders" 
            DL1["base.py"]:::core
            DL2["webvid.py"]:::core
        end
        subgraph "Model Definitions"
            MD1["autoencoder.py"]:::core
            MD2["autoencoder_dualref.py"]:::core
            MD3["ddpm3d.py"]:::core
        end
        subgraph "Samplers"
            S1["ddim.py"]:::core
            S2["ddim_multiplecond.py"]:::core
        end
        subgraph "Diffusion Utilities"
            DU1["basics.py"]:::core
            DU2["distributions.py"]:::core
            DU3["ema.py"]:::core
            DU4["utils_diffusion.py"]:::core
        end
        subgraph "Modules & Networks"
            M1["attention.py"]:::core
            M2["attention_svd.py"]:::core
            M3["condition.py"]:::core
            M4["resampler.py"]:::core
            M5["ae_modules.py"]:::core
            M6["openaimodel3d.py"]:::core
            M7["x_transformer.py"]:::core
        end
    end

    %% Training Pipeline
    subgraph "Training Pipeline (main/)"
        direction TB
        Trainer["trainer.py"]:::pipeline
        Callbacks["callbacks.py"]:::pipeline
        TUtil1["utils_data.py"]:::pipeline
        TUtil2["utils_train.py"]:::pipeline
    end

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        direction TB
        Runner["run.sh"]:::pipeline
        Inference["inference.py"]:::pipeline
        Funcs["funcs.py"]:::pipeline
        DDP["ddp_wrapper.py"]:::pipeline
        VideoSaver["save_video.py"]:::util
    end

    %% Utilities & Assets
    subgraph "Utilities & Assets"
        direction TB
        GeneralUtil["utils.py"]:::util
        Assets["assets/logo"]:::data
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        Checkpoints["Hugging Face Checkpoints<br/>checkpoints/"]:::external
        PyLibs["Python Libs<br/>requirements.txt"]:::external
    end

    %% Connections
    CLI -->|uses| Config
    Gradio -->|uses| Config
    Config -->|feeds hyperparams| Trainer
    Trainer -->|loads data| DL1
    Trainer -->|loads data| DL2
    DL1 -->|provides batches| MD1
    DL2 -->|provides batches| MD3
    MD1 -->|encoded features| M5
    MD2 -->|encoded features| M5
    MD3 -->|3D denoise| S1
    S1 -->|samples| DU1
    S2 -->|samples| DU2
    DU1 -->|applies ema| DU3
    DU4 -->|utility| M6
    Trainer -->|invokes| Callbacks
    Trainer -->|saves| Checkpoints
    Runner -->|loads config| Config
    Runner -->|loads checkpoint| Checkpoints
    Runner -->|calls| MD1
    Runner -->|calls| S1
    Runner -->|calls| DU1
    Runner -->|calls| VideoSaver
    Gradio -->|invokes| Runner
    GeneralUtil -->|helper for| Trainer
    GeneralUtil -->|helper for| Runner
    Assets -->|used in| Gradio

    %% Click Events
    click CLI "https://github.com/doubiiu/tooncrafter/blob/main/scripts/run.sh"
    click Gradio "https://github.com/doubiiu/tooncrafter/blob/main/gradio_app.py"
    click Config "https://github.com/doubiiu/tooncrafter/tree/main/configs/"
    click DL1 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/data/base.py"
    click DL2 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/data/webvid.py"
    click MD1 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/models/autoencoder.py"
    click MD2 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/models/autoencoder_dualref.py"
    click MD3 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/models/ddpm3d.py"
    click S1 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/models/samplers/ddim.py"
    click S2 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/models/samplers/ddim_multiplecond.py"
    click DU1 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/basics.py"
    click DU2 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/distributions.py"
    click DU3 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/ema.py"
    click DU4 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/models/utils_diffusion.py"
    click M1 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/modules/attention.py"
    click M2 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/modules/attention_svd.py"
    click M3 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/modules/encoders/condition.py"
    click M4 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/modules/encoders/resampler.py"
    click M5 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/modules/networks/ae_modules.py"
    click M6 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/modules/networks/openaimodel3d.py"
    click M7 "https://github.com/doubiiu/tooncrafter/blob/main/lvdm/modules/x_transformer.py"
    click Trainer "https://github.com/doubiiu/tooncrafter/blob/main/main/trainer.py"
    click Callbacks "https://github.com/doubiiu/tooncrafter/blob/main/main/callbacks.py"
    click TUtil1 "https://github.com/doubiiu/tooncrafter/blob/main/main/utils_data.py"
    click TUtil2 "https://github.com/doubiiu/tooncrafter/blob/main/main/utils_train.py"
    click Runner "https://github.com/doubiiu/tooncrafter/blob/main/scripts/run.sh"
    click Inference "https://github.com/doubiiu/tooncrafter/blob/main/scripts/evaluation/inference.py"
    click Funcs "https://github.com/doubiiu/tooncrafter/blob/main/scripts/evaluation/funcs.py"
    click DDP "https://github.com/doubiiu/tooncrafter/blob/main/scripts/evaluation/ddp_wrapper.py"
    click VideoSaver "https://github.com/doubiiu/tooncrafter/blob/main/utils/save_video.py"
    click GeneralUtil "https://github.com/doubiiu/tooncrafter/blob/main/utils/utils.py"
    click Assets "https://github.com/doubiiu/tooncrafter/tree/main/assets/logo"
    click Checkpoints "https://github.com/doubiiu/tooncrafter/tree/main/checkpoints/"
    click PyLibs "https://github.com/doubiiu/tooncrafter/blob/main/requirements.txt"

    %% Styles
    classDef core fill:#D0E8FF,stroke:#0366d6,stroke-width:1px
    classDef config fill:#DFFFE0,stroke:#228B22,stroke-width:1px
    classDef frontend fill:#FFF5CC,stroke:#DAA520,stroke-width:1px
    classDef pipeline fill:#FFE8D0,stroke:#CD8500,stroke-width:1px
    classDef util fill:#E8E8E8,stroke:#A9A9A9,stroke-width:1px
    classDef data fill:#F0FFF0,stroke:#2E8B57,stroke-width:1px
    classDef external fill:#F0F0F0,stroke:#808080,stroke-width:1px
```
