How hbb1/2d-gaussian-splatting works
Project Type and Purpose
• This is not a web app or microservice system but a research-oriented Python pipeline for 2D Gaussian Splatting.
• Its main goal is to ingest multi-view scene data (COLMAP or NeRF), train a differentiable renderer (surface elements represented as oriented 2D disks), extract meshes, evaluate quality, and provide both CLI and GUI visualization.
High-Level Component Breakdown
a. Data Ingestion & Preprocessing
• Directories:
• scene/ (colmap_loader.py, dataset_readers.py)
• arguments/ (parses CLI flags)
• Role: load images, camera intrinsics/extrinsics, prepare input structures.
b. Core Training & Rendering Engine
• Modules:
• gaussian_renderer/ (network_gui.py orchestrates forward/backward passes)
• train.py (entry point for training)
• render.py (mesh extraction via TSDF fusion)
• Dependencies:
• submodules/diff-surfel-rasterization (CUDA operator fusing)
• lpipsPyTorch/ (perceptual loss)
• PyTorch as primary DL framework
c. Losses & Metrics
• Files: metrics.py, utils/loss_utils.py
• Role: compute photometric, geometric regularizers (normal consistency, distortion), perceptual losses.
d. Mesh Extraction & TSDF Fusion
• render.py calls Open3D under the hood (via scene/gaussian_model.py) to do bounded/unbounded TSDF fusion.
e. Evaluation Suite
• scripts/ (dtu_eval.py, tnt_eval.py, nerf_eval.py, etc.)
• Each subfolder (eval_dtu, eval_tnt, etc.) contains dataset-specific pipelines: mesh registration, Chamfer/F1 metrics, plotting.
f. Visualization
• view.py (CLI for launching a lightweight viewer)
• gaussian_renderer/network_gui.py (PyQt or similar GUI for monitoring during training)
• External: optional remote/web viewers via submodules or partner repos.
g. Environment & Configuration
• environment.yml (conda spec)
• .gitmodules (pulls in diff-surfel-rasterization and simple-knn)
• scripts/eval_tnt/requirements.txt (pin Open3D version)
Relationships and Data Flow
1. User provides dataset path → arguments module parses flags → scene loaders produce camera/image tensors
2. train.py invokes gaussian_renderer:
• forward/render step uses diff-surfel-rasterization → produces rendered images
• backward step uses losses (metrics.py + lpipsPyTorch) → updates Gaussian parameters
• checkpoint saved to disk
3. render.py loads checkpoint + scene → extracts mesh via TSDF fusion (Open3D) → writes mesh file (PLY)
4. scripts/… evaluation scripts read rendered images or extracted mesh + ground truth → compute metrics → output tables/plots
5. view.py (or network_gui.py) loads model + dataset for interactive inspection
Architectural Patterns & Design Principles
• Modular package structure: scene, utils, metrics, renderer, scripts, submodules
• CLI-driven workflows: each major step (train, render, eval, view) is a separate script
• Separation of concerns:
• Data loaders vs. Core engine vs. Evaluation
• External heavy-compute kernels isolated in submodules
• Plugin-style external libs: lpips already wrapped, diff-surfel fused via submodule
Final Advice
• Keep labels concise but precise: use terms straight from the repo (e.g., “gaussian_renderer”, “render.py”).