# igl-hkust/diffusionasshader architecture

> Architecture diagram of the GitHub repository igl-hkust/diffusionasshader, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/igl-hkust/diffusionasshader
- Repository: https://github.com/igl-hkust/diffusionasshader

## Overview

##### Project Type and Purpose
- The repository implements a 3D-aware video diffusion framework (“Diffusion as Shader”) designed for versatile video generation control. It supports various functions like motion transfer, camera control, object manipulation, and mesh-to-video animation.
- It integrates diffusion models, spatial tracking (via SpaTracker and optionally MoGe), and external tools (like Blender) for generating and controlling video outcomes. It also comes with training and inference pipelines along with a web interface (Gradio-esque via “webui.py”).

##### File Structure
- Top-Level Components:
- Configuration files and submodules: Files like .gitignore, Makefile, pyproject.toml, accelerate_configs/ (with various YAML configurations for distributed training), and requirements.txt.
- Core code directories:
▸ config: For base configurations and parameter definitions.
▸ models: Contains the diffusion, pipelines, and especially the “spatracker” module (deep diving into multiple submodules like dpt, loftr, vit, etc.) that integrates 3D spatial tracking methods.
▸ training and testing: Different scripts and utilities for pre-processing datasets, setting up training arguments, and running inference/evaluation.
▸ scripts: Contains helper scripts for using Blender and launching experiments.
▸ Demo files (demo.py, batch_tracking.py) and the web interface (webui.py) that serve as entry points for inference.
- Architectural insights include the separation between core model definitions, configuration management, training routines, and inference interfaces. The presence of multiple tracking methods (e.g., spatracker, moge) indicates modularity.

##### Instructing the Principal Software Engineer
- Sub-blocks for:
▸ Diffusion Model & Pipelines (including cogvideox and tracking components).
▸ Spatial Tracking Module (with its submodules: dpt, loftr, vit, unet).
▸ Data/Configuration Management.
▸ Inference (demo.py, webui.py) and Training Scripts (training directory).
▸ For example, data flows from the “User Interface” to “Preprocessing” → “Diffusion & Tracking Models” → “Output”.
▸ Core processing modules (e.g., models, tracking) in another, and so on.
- To incorporate annotations that call out key external dependencies (PyTorch, Accelerate, Blender) and any architectural decision like “modularity” and “distributed training.”

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface & Inference Layer
    subgraph "User Interface & Inference"
        UI1["demo.py"]:::interface
        UI2["webui.py"]:::interface
        UI3["batch_tracking.py"]:::interface
    end

    %% Data and Input Management
    subgraph "Data and Input Management"
        DATA1["Assets & Dataset"]:::data
        DATA2["Dataset Preparation"]:::data
    end

    %% Model & Core Processing
    subgraph "Model & Core Processing"
        MODEL1["Diffusion Model"]:::model
        MODEL2["Pipeline"]:::model
        MODEL3["Spatial Tracking Module"]:::model
    end

    %% Training Infrastructure
    subgraph "Training Infrastructure"
        TRAIN1["Training Routines"]:::training
        TRAIN2["Accelerate Configs"]:::training
    end

    %% Configuration & Utilities
    subgraph "Configuration & Utilities"
        UTIL1["Configuration Management"]:::utility
        UTIL2["Helper Utilities"]:::utility
        UTIL3["Accelerate Tracking Utility"]:::utility
    end

    %% External Dependencies
    subgraph "External Dependencies"
        EXT1["PyTorch"]:::external
        EXT2["Accelerate"]:::external
        EXT3["Blender"]:::external
    end

    %% Connections from User Interface to Model Processing
    UI1 -->|"triggers"| MODEL2
    UI2 -->|"triggers"| MODEL2
    UI3 -->|"utilizes"| MODEL3

    %% Data Flow from Assets to Dataset Preparation and Training
    DATA1 -->|"provides"| DATA2
    DATA2 -->|"prepares_data_for"| TRAIN1

    %% Training updates the Model Components
    TRAIN1 -->|"updates"| MODEL1
    TRAIN1 -->|"updates"| MODEL2

    %% Configuration drives various components
    UTIL1 -->|"configures"| UI1
    UTIL1 -->|"configures"| UI2
    UTIL1 -->|"configures"| TRAIN1
    UTIL1 -->|"configures"| MODEL1
    UTIL1 -->|"configures"| MODEL2

    %% Helper Utilities support Model and Training
    UTIL2 -->|"assists"| MODEL3
    UTIL2 -->|"assists"| TRAIN1

    %% Additional Utility supports Training
    UTIL3 -->|"supports"| TRAIN1

    %% Accelerate Configs orchestrate Training routines
    TRAIN2 -->|"orchestrates"| TRAIN1

    %% External Dependencies interactions
    MODEL2 -->|"runs_on"| EXT1
    TRAIN1 -->|"distributed_with"| EXT2
    UTIL2 -->|"calls"| EXT3

    %% Click Events for Components
    click UI1 "https://github.com/igl-hkust/diffusionasshader/blob/main/demo.py"
    click UI2 "https://github.com/igl-hkust/diffusionasshader/blob/main/webui.py"
    click UI3 "https://github.com/igl-hkust/diffusionasshader/blob/main/batch_tracking.py"
    click DATA1 "https://github.com/igl-hkust/diffusionasshader/tree/main/assets/"
    click DATA2 "https://github.com/igl-hkust/diffusionasshader/blob/main/training/prepare_dataset.py"
    click MODEL1 "https://github.com/igl-hkust/diffusionasshader/blob/main/models/cogvideox_tracking.py"
    click MODEL2 "https://github.com/igl-hkust/diffusionasshader/blob/main/models/pipelines.py"
    click MODEL3 "https://github.com/igl-hkust/diffusionasshader/tree/main/models/spatracker/"
    click TRAIN1 "https://github.com/igl-hkust/diffusionasshader/tree/main/training/"
    click TRAIN2 "https://github.com/igl-hkust/diffusionasshader/tree/main/accelerate_configs/"
    click UTIL1 "https://github.com/igl-hkust/diffusionasshader/tree/main/config/"
    click UTIL2 "https://github.com/igl-hkust/diffusionasshader/tree/main/scripts/"
    click UTIL3 "https://github.com/igl-hkust/diffusionasshader/blob/main/accelerate_tracking.py"

    %% Styles
    classDef interface fill:#add8e6,stroke:#000,stroke-width:2px;
    classDef data fill:#f4a460,stroke:#000,stroke-width:2px;
    classDef model fill:#90ee90,stroke:#000,stroke-width:2px;
    classDef training fill:#ffff99,stroke:#000,stroke-width:2px;
    classDef utility fill:#dda0dd,stroke:#000,stroke-width:2px;
    classDef external fill:#ffcccb,stroke:#000,stroke-width:2px;
```
