How igl-hkust/diffusionasshader works
Project Type and Purpose
• The repository implements a 3D-aware video diffusion framework (“Diffusion as Shader”) designed for versatile video generation control. It supports various functions like motion transfer, camera control, object manipulation, and mesh-to-video animation.
• It integrates diffusion models, spatial tracking (via SpaTracker and optionally MoGe), and external tools (like Blender) for generating and controlling video outcomes. It also comes with training and inference pipelines along with a web interface (Gradio-esque via “webui.py”).
File Structure
• Top-Level Components:
• Configuration files and submodules: Files like .gitignore, Makefile, pyproject.toml, accelerate_configs/ (with various YAML configurations for distributed training), and requirements.txt.
• Core code directories:
▸ config: For base configurations and parameter definitions.
▸ models: Contains the diffusion, pipelines, and especially the “spatracker” module (deep diving into multiple submodules like dpt, loftr, vit, etc.) that integrates 3D spatial tracking methods.
▸ training and testing: Different scripts and utilities for pre-processing datasets, setting up training arguments, and running inference/evaluation.
▸ scripts: Contains helper scripts for using Blender and launching experiments.
▸ Demo files (demo.py, batch_tracking.py) and the web interface (webui.py) that serve as entry points for inference.
• Architectural insights include the separation between core model definitions, configuration management, training routines, and inference interfaces. The presence of multiple tracking methods (e.g., spatracker, moge) indicates modularity.
Instructing the Principal Software Engineer
• Sub-blocks for:
▸ Diffusion Model & Pipelines (including cogvideox and tracking components).
▸ Spatial Tracking Module (with its submodules: dpt, loftr, vit, unet).
▸ Data/Configuration Management.
▸ Inference (demo.py, webui.py) and Training Scripts (training directory).
▸ For example, data flows from the “User Interface” to “Preprocessing” → “Diffusion & Tracking Models” → “Output”.
▸ Core processing modules (e.g., models, tracking) in another, and so on.
• To incorporate annotations that call out key external dependencies (PyTorch, Accelerate, Blender) and any architectural decision like “modularity” and “distributed training.”