How yzslab/gaussian-splatting-lightning works
Project Type and Purpose
• It is not a typical full-stack web application but rather a research/development system focused on 3D scene reconstruction with advanced rendering techniques (e.g., Gaussian splatting, deformable models, mip-splatting, appearance embedding, etc.).
• Its purpose is to provide multi-GPU training, support a variety of datasets (Blender, Colmap, PolyCam, etc.), and to deliver diverse rendering pipelines (including a web-based viewer). The README makes it clear that the system supports training routines, evaluation, and interactive viewing.
Analyzing the File Structure
• Key top-level directories like “configs”, “internal”, “utils”, “notebooks” and “tests” reveal a modular, layered design.
• Under “internal”, you have subdirectories for critical components:
• “dataparsers”: Handles various dataset inputs (e.g., Colmap, Blender, NERVF datasets).
• “models” and “model_components”: Contain the core Gaussian models and their variants (e.g., vanilla, deformable, appearance models).
• “renderers”: Implements multiple rendering strategies (e.g., gsplat, vanilla, deformable, distributed renderers, etc.).
• “density_controllers”: Contains modules to manage density, including specialized controllers like MCMC, static, or taming-based ones.
• “configs”: Contains Python scripts to interpret and manage training configuration.
• “utils”: Houses scripts for dataset preprocessing, conversion (e.g., ckpt2ply, image downsampling), merging model partitions, etc.
• “viewer”: Provides an interactive web view layer powered by a web viewer library “viser” for model visualization.
• “main.py” and “render.py” are the entry points for training and rendering flows.
• The “notebooks” folder shows several exploratory and preprocessing pipelines, while “tests” provide unit tests for core functionality.
Key Architectural Components
a. Data Handling & Preprocessing
• Data input is managed by multiple dataparsers (located in internal/dataparsers) that support different dataset types (Colmap, Blender, PolyCam, etc.).
• Preprocessing scripts (in utils) handle operations like downsampling images, generating appearance groups, estimating depths, and converting models (e.g., ckpt2ply).
b. Configuration / Hyperparameter Management
• A large collection of YAML config files in the “configs” folder (with various sub-categories) that dictate training parameters, model architecture, rendering options, and multi-GPU settings.
• Internal configuration modules (found in internal/configs and utils/scalable_param_configs) allow dynamic adjustment of hyperparameters.
c. Model Components & Rendering Pipelines
• “internal/models” and “internal/model_components” incorporate various Gaussian splatting methods (basic Gaussian, deformable, periodic vibration, etc.).
• “internal/renderers” contains distinct renderers that implement different strategies:
• Standard GSplat renderers
• Appearance embedding renderers (both view-dependent and independent versions)
• Distributed renderers for multi-GPU training
• Variants for specific derived algorithms (Mip-Splatting, Feature Distillation, etc.)
• Density controllers (internal/density_controllers) modulate the evolution and densification of Gaussians during training.
d. Training & Optimization
• The training pipeline is orchestrated from main.py, which integrates:
• PyTorch Lightning for scheduled training loops.
• Multi-GPU strategies (both standard DDP and a newer distributed approach for densification).
• Logging, callbacks, and metric computations (as seen in internal/metrics).
e. Post-Processing & Evaluation
• Tools for converting checkpoints to ply, merging distributed checkpoints (utils/merge_distributed_ckpts.py), and for rendering videos.
• Evaluation modules in main.py (and separate commands) to compute image-based metrics (PSNR, SSIM, LPIPS).
f. Interactive Visualization
• The “viewer” module (internal/viewer and viewer.py) provides a web-based interactive UI allowing users to load one or more models, adjust camera transforms, and view results in real-time.
Relationships and Data Flow
• Start at the Data Layer: The dataset is preprocessed (via utils or notebooks) and loaded using a specific dataparser.
• The parsed data is then fed into the Training Pipeline (main.py) where configurations from YAML files determine which model components and density controllers to instantiate.
• The results (trained checkpoints) may be post-processed (eg, converted to ply) and optionally loaded in the web viewer.
• Multi-GPU modules alter data loading (distributing subsets to each process) and coordinate synchronized updates via Lightning’s DDP or custom strategies.