How nju-3dv/relightable3dgaussian works
Main System Type and Goals
• This is a research-oriented, full-stack 3D rendering and relighting framework.
• Key purposes:
• Train and evaluate real-time point-cloud BRDF models using PyTorch.
• Accelerate ray tracing via custom CUDA extensions (BVH, knn, rasterizer).
• Prepare datasets (NeRF, DTU, Tanks & Temples) with VisMVSNet.
• Provide a GUI for live visualization and a suite of CLI scripts for training, evaluation, and relighting.
Break Down Top-Level Components
• User Interface / CLI Layer
• train.py, relighting.py, eval_nvs.py, eval_relighting_syn4.py, gui.py, convert.py, script/*.sh
• Data & Dataset Preparation
• configs/ (scene trajectories, transforms)
• scene/ package (dataset_readers, colmap_loader, envmap handlers)
• vismvsnet/ submodule (depth-map generation, preprocessing scripts)
• Core Python Pipeline
• gaussian_renderer/ (model definitions, composite rendering logic)
• utils/ (camera, graphics, image, loss, system utilities)
• lpipsPyTorch/ (perceptual loss modules)
• Native Extensions & Acceleration
• submodules/simple-knn (CUDA K-NN nearest neighbor)
• bvh (CUDA BVH construction and tracing)
• r3dg-rasterization (CUDA rasterizer for 3D Gaussian splats)
• External: nvdiffrast, torch_scatter, kornia
• Storage & Artifacts
• datasets/ (organized by NeRF_syn, neilfpp, Synthetic4Relight)
• media/, env_map/ (HDR environment maps)
• output/ (model checkpoints, rendered frames, evaluation metrics)
Sketch Relationships and Data Flows
• CLI scripts invoke the Python “Controller” layer.
• Controller calls Data Preparation modules (scene + vismvsnet) to load or preprocess inputs.
• Preprocessed data feeds the Training Engine (train.py → gaussian_renderer + utils).
• Training Engine offloads heavy ops to GPU via:
• BVH module for acceleration of ray tracing queries
• simple-knn for point-cloud neighbor searches
• r3dg-rasterization for forward/backward splatting operations
• Trained models and intermediate outputs are saved to disk.
• Evaluation scripts load checkpoints and rerun render/eval routines with the same pipeline.
• GUI attaches to a running or saved model directory for interactive camera control and rendering.
• relighting.py orchestrates multi-object composition by combining point clouds from different runs and environment maps.
Architectural Patterns & Principles
• Layered architecture: CLI → Python controller → Core modules → Native extensions → GPU hardware
• Plugin/extension pattern: custom CUDA code is packaged as pip-installable modules and imported seamlessly into Python.
• Separation of concerns: data I/O (scene, vismvsnet), model logic (gaussian_renderer), utilities, and acceleration code.
• Reuse of third-party libraries for specialized tasks (e.g., kornia for normals, LPIPS for perceptual loss).