How doubiiu/tooncrafter works
Project type and purpose
• ToonCrafter is an open-source research tool for generative video interpolation of cartoon frames. It provides:
• Training pipeline (configs, main/trainer, lvdm modules)
• Inference pipeline (CLI scripts, Gradio app)
• Core diffusion model library (lvdm/)
• Utilities for data loading, evaluation, saving videos
File structure
• Top-level directories:
• configs/ – YAML configs and run.sh for different resolutions
• lvdm/ – core diffusion library: data loaders, model definitions, samplers, modules
• main/ – training entrypoint, callbacks, utilities
• scripts/ – evaluation, inference, Gradio demos, run.sh
• utils/, prompts/, assets/
• gradio_app.py – standalone UI
• Checkpoints and external model weights live under checkpoints/ (created at inference)
• requirements.txt, LICENSE, README.md
README for additional insights
• Describes use cases: cartoon interpolation, sketch guidance, sketch colorization
• Details on model resolution, GPU memory, links to Hugging Face checkpoints
• Instructions for CLI inference (scripts/run.sh) and Gradio demo
a. Main components
1. User Interfaces
– CLI wrapper (scripts/run.sh → scripts/evaluation/inference.py)
– Local Gradio UI (gradio_app.py + scripts/gradio/)
2. Configuration
– configs/*.yaml & run.sh
3. Core Library (lvdm/)
– Data loaders (lvdm/data/)
– Model definitions (lvdm/models/, samplers)
– Modules & networks (lvdm/modules/)
– Diffusion utilities (lvdm/basics.py, distributions.py, ema.py)
4. Training Pipeline (main/)
– Trainer (main/trainer.py)
– Callbacks (main/callbacks.py)
– Data & training utilities (main/utils_*.py)
5. Inference Pipeline
– Scripts (scripts/evaluation/inference.py)
– Video saving (utils/save_video.py)
6. External Dependencies
– Pretrained checkpoints (Hugging Face)
– Python libs (requirements.txt)
b. Relationships & interactions
• User → CLI or UI → loads config → loads model checkpoint → calls lvdm core to run diffusion inference → outputs video → utils/save_video
• Training: main/trainer reads configs → data loaders → lvdm model → training loop → callbacks → save checkpoint
• Scripts import lvdm modules and utils
c. Architectural patterns & design principles
• Modular separation: core library vs. application scripts
• Config-driven design: all hyperparameters in configs/
• Clear split between training and inference code paths
• Extensibility: additional samplers, modules, guidance easily plugged in under lvdm/
d. Key technologies & libraries
• PyTorch (models, training loop)
• YAML configs (Hydra-style)
• Gradio (UI)
• FFmpeg or OpenCV (video saving)
• Shell scripts for orchestration
Detailed breakdown of components
• Config manager
• Data loader
• Model encoder/decoder modules
• Sampler (DDIM, multi-cond)
• EMA & diffusion utilities
• Training orchestrator (trainer + callbacks)
• Inference engine (script + video saver)
• CLI & Gradio frontends
• Checkpoint store & assets