# nv-tlabs/diffusion-renderer architecture

> Architecture diagram of the GitHub repository nv-tlabs/diffusion-renderer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nv-tlabs/diffusion-renderer
- Repository: https://github.com/nv-tlabs/diffusion-renderer

## Overview

##### Project Type and Purpose
- DiffusionRenderer is a full-stack research framework for video inverse (delighting) and forward (relighting) rendering powered by diffusion-based neural networks. It provides command-line inference scripts, a modular codebase (data, models, pipelines), and relies on PyTorch, OmegaConf, and Hugging Face for model checkpoints.

##### File Structure Analysis
- Top level
- configs/ (rgbx_inference.yaml, xrgb_inference.yaml) – YAML configs driving the pipelines via OmegaConf
- examples/
– hdri/ (HDR panoramas)
– videos/ (sample mp4s)
- inference_svd_rgbx.py & inference_svd_xrgb.py – CLI entry-point scripts
- requirements.txt, LICENSE, README.md
- src/
- data/ (rendering_utils.py: I/O, transformations)
- models/ (custom_u-net architectures, env encoder, condition modules)
- pipelines/ (pipeline_rgbx.py & pipeline_xrgb.py: orchestrate preprocessing, model inference, post-processing)
- utils/ (frame extraction, weight download, RGBX utilities)

##### Key Components
- CLI Entry Points (inference_svd_rgbx.py, inference_svd_xrgb.py)
- Configuration Manager (OmegaConf + YAML)
- Data Ingestion & Preprocessing (utils/dataproc_extract_frames_from_video.py, src/data/rendering_utils.py)
- Model Layer (src/models/*: CustomUNet_ST, EnvEncoder, UnetCondition)
- Pipeline Orchestrator (src/pipelines/*: load config, fetch weights, run model, save outputs)
- Utilities (utils/download_weights.py, utils/utils_rgbx*.py)
- External Services (Hugging Face model registry)
- Hardware (GPU cluster for accelerated inference)
- File System (input/output directories, examples/, configs/)

##### Relationships & Data Flow
1. User invokes CLI script → parses CLI args & loads YAML config
2. Pipeline class instantiated with config
3. Pipeline calls download_weights (Hugging Face) → loads model into PyTorch on GPU
4. Pipeline loads video frames via rendering_utils or frame-extractor
5. Preprocessed data passed to diffusion model (inverse or forward)
6. Model outputs scene attributes or relighted frames
7. Post-processing via utils_rgbx → writes frames/videos to disk
8. Optional: frames reassembled into MP4

##### Architectural Patterns & Design Principles
- Layered Modular Structure:
CLI → Config → Pipeline → Model → Data → Utils → I/O
- Separation of Concerns: distinct modules for orchestration, model definitions, data handling, utilities
- Extensibility: easy to swap models or adjust pipelines via src/models and src/pipelines
- Reusability: shared utility functions and data abstractions
- GPU-accelerated, with optional distributed inference

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI & Config Layer
    subgraph "CLI & Config Layer"
        CLI1["inference_svd_rgbx.py"]:::cli
        click CLI1 "https://github.com/nv-tlabs/diffusion-renderer/blob/main/inference_svd_rgbx.py"
        CLI2["inference_svd_xrgb.py"]:::cli
        click CLI2 "https://github.com/nv-tlabs/diffusion-renderer/blob/main/inference_svd_xrgb.py"
        Config1["rgbx_inference.yaml"]:::config
        click Config1 "https://github.com/nv-tlabs/diffusion-renderer/blob/main/configs/rgbx_inference.yaml"
        Config2["xrgb_inference.yaml"]:::config
        click Config2 "https://github.com/nv-tlabs/diffusion-renderer/blob/main/configs/xrgb_inference.yaml"
    end

    %% Pipeline Layer
    subgraph "Pipeline Layer"
        Pipeline["Pipeline Orchestrator\n(src/pipelines/pipeline_rgbx.py)"]:::pipeline
        click Pipeline "https://github.com/nv-tlabs/diffusion-renderer/blob/main/src/pipelines/pipeline_rgbx.py"
    end

    %% Data & Utility Layer
    subgraph "Data & Utility Layer"
        DataLayer["Data Layer\n(Frame Extractor, rendering_utils)"]:::data
        click DataLayer "https://github.com/nv-tlabs/diffusion-renderer/blob/main/utils/dataproc_extract_frames_from_video.py"
        click DataLayer "https://github.com/nv-tlabs/diffusion-renderer/blob/main/src/data/rendering_utils.py"
        UtilLayer["Utility Layer"]:::utils
        click UtilLayer "https://github.com/nv-tlabs/diffusion-renderer/blob/main/utils/download_weights.py"
        click UtilLayer "https://github.com/nv-tlabs/diffusion-renderer/blob/main/utils/utils_env_proj.py"
        click UtilLayer "https://github.com/nv-tlabs/diffusion-renderer/blob/main/utils/utils_rgbx.py"
        click UtilLayer "https://github.com/nv-tlabs/diffusion-renderer/blob/main/utils/utils_rgbx_inference.py"
    end

    %% Model Layer
    subgraph "Model Layer"
        Model1["CustomUNet_ST"]:::model
        click Model1 "https://github.com/nv-tlabs/diffusion-renderer/blob/main/src/models/custom_unet_st.py"
        Model2["CustomUNet_ST_Blocks"]:::model
        click Model2 "https://github.com/nv-tlabs/diffusion-renderer/blob/main/src/models/custom_unet_st_blocks.py"
        Model3["EnvEncoder"]:::model
        click Model3 "https://github.com/nv-tlabs/diffusion-renderer/blob/main/src/models/env_encoder.py"
        Model4["UnetCondition"]:::model
        click Model4 "https://github.com/nv-tlabs/diffusion-renderer/blob/main/src/models/unet_condition.py"
    end

    %% External & Hardware
    External["Hugging Face Registry"]:::external
    Hardware["GPU Cluster"]:::hardware

    %% Storage Layer
    StorageIn["Input Examples"]:::storage
    click StorageIn "https://github.com/nv-tlabs/diffusion-renderer/tree/main/examples/hdri/"
    click StorageIn "https://github.com/nv-tlabs/diffusion-renderer/tree/main/examples/videos/"
    StorageOut["Output Directory"]:::storage

    %% Connections
    CLI1 -->|"loads config"| Config1
    CLI2 -->|"loads config"| Config2
    Config1 -->|"init pipeline"| Pipeline
    Config2 -->|"init pipeline"| Pipeline
    StorageIn -->|"provides data"| DataLayer
    Pipeline -->|"calls"| DataLayer
    Pipeline -->|"uses"| UtilLayer
    Pipeline -->|"loads model"| Model1
    Pipeline -->|"loads model"| Model2
    Pipeline -->|"loads model"| Model3
    Pipeline -->|"loads model"| Model4
    Pipeline -->|"downloads weights"| External
    Pipeline -->|"executes on"| Hardware
    Pipeline -->|"writes results"| StorageOut

    %% Styles
    classDef cli fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef config fill:#87CEFA,stroke:#333,stroke-width:1px
    classDef pipeline fill:#FFA500,stroke:#333,stroke-width:1px
    classDef data fill:#98FB98,stroke:#333,stroke-width:1px
    classDef model fill:#FFD700,stroke:#333,stroke-width:1px
    classDef utils fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef external fill:#DA70D6,stroke:#333,stroke-width:1px
    classDef hardware fill:#FF6347,stroke:#333,stroke-width:1px
    classDef storage fill:#40E0D0,stroke:#333,stroke-width:1px
```
