# nju-3dv/relightable3dgaussian architecture

> Architecture diagram of the GitHub repository nju-3dv/relightable3dgaussian, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nju-3dv/relightable3dgaussian
- Repository: https://github.com/nju-3dv/relightable3dgaussian

## Overview

##### Main System Type and Goals
- This is a research-oriented, full-stack 3D rendering and relighting framework.
- Key purposes:
- Train and evaluate real-time point-cloud BRDF models using PyTorch.
- Accelerate ray tracing via custom CUDA extensions (BVH, knn, rasterizer).
- Prepare datasets (NeRF, DTU, Tanks & Temples) with VisMVSNet.
- Provide a GUI for live visualization and a suite of CLI scripts for training, evaluation, and relighting.

##### Break Down Top-Level Components
- User Interface / CLI Layer
- train.py, relighting.py, eval_nvs.py, eval_relighting_syn4.py, gui.py, convert.py, script/*.sh
- Data & Dataset Preparation
- configs/ (scene trajectories, transforms)
- scene/ package (dataset_readers, colmap_loader, envmap handlers)
- vismvsnet/ submodule (depth-map generation, preprocessing scripts)
- Core Python Pipeline
- gaussian_renderer/ (model definitions, composite rendering logic)
- utils/ (camera, graphics, image, loss, system utilities)
- lpipsPyTorch/ (perceptual loss modules)
- Native Extensions & Acceleration
- submodules/simple-knn (CUDA K-NN nearest neighbor)
- bvh (CUDA BVH construction and tracing)
- r3dg-rasterization (CUDA rasterizer for 3D Gaussian splats)
- External: nvdiffrast, torch_scatter, kornia
- Storage & Artifacts
- datasets/ (organized by NeRF_syn, neilfpp, Synthetic4Relight)
- media/, env_map/ (HDR environment maps)
- output/ (model checkpoints, rendered frames, evaluation metrics)

##### Sketch Relationships and Data Flows
- CLI scripts invoke the Python “Controller” layer.
- Controller calls Data Preparation modules (scene + vismvsnet) to load or preprocess inputs.
- Preprocessed data feeds the Training Engine (train.py → gaussian_renderer + utils).
- Training Engine offloads heavy ops to GPU via:
- BVH module for acceleration of ray tracing queries
- simple-knn for point-cloud neighbor searches
- r3dg-rasterization for forward/backward splatting operations
- Trained models and intermediate outputs are saved to disk.
- Evaluation scripts load checkpoints and rerun render/eval routines with the same pipeline.
- GUI attaches to a running or saved model directory for interactive camera control and rendering.
- relighting.py orchestrates multi-object composition by combining point clouds from different runs and environment maps.

##### Architectural Patterns & Principles
- Layered architecture: CLI → Python controller → Core modules → Native extensions → GPU hardware
- Plugin/extension pattern: custom CUDA code is packaged as pip-installable modules and imported seamlessly into Python.
- Separation of concerns: data I/O (scene, vismvsnet), model logic (gaussian_renderer), utilities, and acceleration code.
- Reuse of third-party libraries for specialized tasks (e.g., kornia for normals, LPIPS for perceptual loss).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface / CLI Layer
    subgraph "User Interface / CLI Layer"
        direction TB
        train["train.py"]:::python
        relight["relighting.py"]:::python
        evalNVS["eval_nvs.py"]:::python
        evalRelight["eval_relighting_syn4.py"]:::python
        gui["gui.py"]:::python
        convert["convert.py"]:::python
        scripts["script/"]:::python
    end

    %% Data & Dataset Preparation
    subgraph "Data & Dataset Preparation"
        direction TB
        configs["configs/"]:::datastore
        scene["scene/"]:::python
        mvsnet["vismvsnet/"]:::python
    end

    %% Core Python Pipeline
    subgraph "Core Python Pipeline"
        direction TB
        gr["gaussian_renderer/"]:::python
        utils["utils/"]:::python
        lpips["lpipsPyTorch/"]:::python
    end

    %% Native Extensions & Acceleration
    subgraph "Native Extensions & Acceleration"
        direction TB
        bvhMod["bvh/"]:::native
        knn["submodules/simple-knn/"]:::native
        raster["r3dg-rasterization/"]:::native
    end

    %% Storage & Artifacts
    subgraph "Storage & Artifacts"
        direction TB
        envmap["env_map/"]:::datastore
        media["media/"]:::datastore
        output["output/"]:::datastore
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        pytorch["PyTorch"]:::external
        conda["Conda"]:::external
        gpu["NVIDIA GPU"]:::external
        nvdiffrast["nvdiffrast"]:::external
        torchscatter["torch_scatter"]:::external
        kornia["kornia"]:::external
    end

    %% Relationships
    train -->|invokes| configs
    train -->|invokes| scene
    train -->|invokes| mvsnet
    relight -->|uses data from| scene
    relight -->|uses maps from| envmap
    evalNVS -->|loads checkpoint| gr
    evalRelight -->|loads checkpoint| gr
    gui -->|interactive render| gr
    convert -->|processes HDR| envmap
    scripts -->|batch pipelines| train

    configs -->|provides transforms| gr
    scene -->|loads datasets| gr
    mvsnet -->|preprocesses depth| gr

    gr -->|imports utils| utils
    gr -->|imports perceptual loss| lpips

    gr -->|calls| bvhMod
    gr -->|calls| knn
    gr -->|calls| raster

    gr -->|saves outputs| output

    relight -->|saves relit scenes| output
    gui -->|writes logs| output

    envmap -->|provides maps| relight
    media -->|assets for GUI| gui

    %% External integration
    gr -->|runs on| gpu
    gr -->|depends on| pytorch
    gr -->|uses| nvdiffrast
    gr -->|uses| torchscatter
    gr -->|uses| kornia
    gr -->|environment via| conda

    %% Click Events
    click train "https://github.com/nju-3dv/relightable3dgaussian/blob/main/train.py"
    click relight "https://github.com/nju-3dv/relightable3dgaussian/blob/main/relighting.py"
    click evalNVS "https://github.com/nju-3dv/relightable3dgaussian/blob/main/eval_nvs.py"
    click evalRelight "https://github.com/nju-3dv/relightable3dgaussian/blob/main/eval_relighting_syn4.py"
    click gui "https://github.com/nju-3dv/relightable3dgaussian/blob/main/gui.py"
    click convert "https://github.com/nju-3dv/relightable3dgaussian/blob/main/convert.py"
    click scripts "https://github.com/nju-3dv/relightable3dgaussian/tree/main/script/"
    click configs "https://github.com/nju-3dv/relightable3dgaussian/tree/main/configs/"
    click scene "https://github.com/nju-3dv/relightable3dgaussian/tree/main/scene/"
    click mvsnet "https://github.com/nju-3dv/relightable3dgaussian/tree/main/vismvsnet/"
    click gr "https://github.com/nju-3dv/relightable3dgaussian/tree/main/gaussian_renderer/"
    click utils "https://github.com/nju-3dv/relightable3dgaussian/tree/main/utils/"
    click lpips "https://github.com/nju-3dv/relightable3dgaussian/tree/main/lpipsPyTorch/"
    click bvhMod "https://github.com/nju-3dv/relightable3dgaussian/tree/main/bvh/"
    click knn "https://github.com/nju-3dv/relightable3dgaussian/tree/main/submodules/simple-knn/"
    click raster "https://github.com/nju-3dv/relightable3dgaussian/tree/main/r3dg-rasterization/"
    click envmap "https://github.com/nju-3dv/relightable3dgaussian/tree/main/env_map/"
    click media "https://github.com/nju-3dv/relightable3dgaussian/tree/main/media/"

    %% Styles
    classDef python fill:#B3CDE3,stroke:#03396C,stroke-width:1px
    classDef native fill:#FBB4AE,stroke:#B22222,stroke-width:1px
    classDef datastore fill:#C7E9C0,stroke:#238B45,stroke-width:1px
    classDef external fill:#DDDDDD,stroke:#888888,stroke-width:1px
```
