How city-super/scaffold-gs works
Major architectural layers and modules
• Data Ingestion & Preprocessing:
– Raw input data (images, COLMAP outputs, SfM point-cloud data) organized under data/ – see the “Data” section in the README.
– Preprocessing tools (scripts in “dataset_tools” and various converters in SIBR_viewers’s documentation for Colmap, etc.).
• Core Training Pipeline:
– The Python training scripts (e.g. train.py, train_xxx.sh, single_train.sh) initiate the learning process.
– Submodules like “diff-gaussian-rasterization” and “simple-knn” (in the submodules folder) indicate that CUDA-level rasterization and spatial data handling are used to accelerate rendering.
• Model Checkpointing & Configuration:
– There are configuration files and arguments (cfg_args) that control training (and even a system to update anchor points via voxel_size and update_init_factor parameters).
• Rendering & Evaluation Pipeline:
– The training code automatically triggers evaluation procedures: rendering results (using the integrated diff‐Gaussian rendering), fps estimation and quality metrics printed/logged.
– In addition, utilities such as render.py and metrics.py provide manual rendering and metric computations.
– Under “gaussian_renderer” and “scene” folders, additional tools for setting up the rendering scene exist.
• Interactive Viewer / SIBR_viewers:
– The “SIBR_viewers” directory contains the C++/CMake code, assets, and configuration to build an interactive viewer. This component is used for visualizing the final trained model thereby providing real‐time inspection.
Relationships and interactions
• Data Flow: Raw input data (images and SfM results) are preprocessed and fed into the training module. The training module uses these inputs to build the scene representation using an explicit anchor-based system (structured 3D Gaussians), learning per-anchor attributes with various MLP components.
• Between Training and Rendering: Once the training is complete, the checkpoint (point cloud and network weights) is used by the rendering pipeline (both automated evaluation in training code and the interactive viewer component).
• The rendering pipeline makes calls to low-level CUDA modules (diff-gaussian-rasterization) for fast rasterization and rendering computations.
• Metrics and quality evaluation are executed after rendering to compute metrics (e.g., error, fps) – these outcomes are logged.
Technical ecosystem and design patterns
• Layer Separation: Clearly separate data ingestion/preprocessing, training (and model optimization), rendering, and viewing.
• Heterogeneous implementation: The project is hybrid – a mix of C++/CUDA components (for performance rendering, viewer, and some core functionalities in SIBR_viewers) and Python modules (for training, evaluation, and utility scripts). Represent how the C++ and Python components interact (through saved model checkpoints, file I/O, and configuration files).
• External dependencies: Incorporate external libraries like Colmap (for SfM processing), LPIPS PyTorch (for perceptual loss metrics), and the third-party diff-gaussian-rasterization module.
1. Data Input & Preprocessing: “Input Data (Images, SfM/Colmap Outputs)”, “Dataset Tools & Converters”
3. Model Storage: “Checkpoint Files (point_cloud.ply, network weights .pt, cfg_args, cameras.json)”
4. Rendering & Evaluation: “Rendering Engine (integrating diff-gaussian-rasterization, evaluation modules: render.py, metrics.py)”
5. Viewing: “Interactive Viewer (SIBR_viewers – C++/CMake based interactive display)”
• Annotations like “CUDA acceleration”, “MLP-based attribute prediction”, and “External Library Dependencies (e.g., Colmap, LPIPS)”.