How kaikai23/maskgaussian works
Project Type and Purpose
• The project is a computer vision / graphics application aimed at improving rendering speed, reducing GPU memory usage, and enabling both from-scratch training and post-training pruning for 3D Gaussian representations.
• It is built on top of existing 3D-GS frameworks and integrates probabilistic mask-based approaches to adaptively prune Gaussians.
Analyzing the File Structure
• Top-level modules:
• Main scripts and executables: “run_all.py”, “train.py”, “render.py”, “metrics.py”, “prune_finetune.py”
• Core modules grouped by functionality: “scene”, “gaussian_renderer”, “arguments”, “lpipsPyTorch”, “utils”
• Supporting submodules: located in the “submodules” directory (e.g., “mask-diff-gaussian-rasterization”, “diff-gaussian-rasterization”, “simple-knn”)
• Assets and environment configurations: “assets”, “environment.yml”, and configuration files like “.gitmodules” and “.gitignore”
• The overall architecture hints at a pipeline where data flows from dataset preparation (external “data/” folder) into the training pipeline, gets processed by various scene and rendering modules, and finally gets evaluated and pruned.
Key Architectural Components
a. Data Preparation & Input:
• External “data/” folder (organized into scenes with images and sparse views)
• Data readers in the “scene/dataset_readers.py” (and possibly “scene/colmap_loader.py”) for handling camera parameters and SfM data
b. Training Pipeline:
• Command scripts like “run_all.py” that tie together training, rendering, and evaluation
• “train.py” for single-scene training with arguments to tweak loss parameters (for instance, mask loss start and end iterations)
• “prune_finetune.py” and “scripts/run_prune_finetune.sh” to represent the post-training pruning procedure
c. Rendering & Visualization:
• “gaussian_renderer” module to include rendering functionalities (notably “network_gui.py” which might offer a GUI-based interface)
• “render.py” for producing final renderings usable with popular 3D viewers (e.g., SuperSplat or antimatter15)
d. Losses and Evaluation:
• “lpipsPyTorch” module for perceptual loss measurements (LPIPS), indicating integration with deep learning frameworks (PyTorch specifically)
• “metrics.py” for quantitative evaluation of the rendered outputs
e. Utility and Helper Functions:
• “arguments” module to parse command-line inputs
• “utils” directory providing support with camera utilities, graphics utilities, and shell/system helpers
f. External Dependencies / Submodules:
• That submodules like “mask-diff-gaussian-rasterization”, “diff-gaussian-rasterization”, and “simple-knn” are integrated as crucial external components to enable differentiable rasterization and related operations.
Relationships and Data Flow
• A node for the Training Process (train.py / run_all.py), which then interacts with sub-components:
◦ Scene module handling 3D Gaussian representations (scene/gaussian_model, scene/gaussian_model_3dgs)
◦ The Rendering module (“gaussian_renderer”)
◦ Loss computation using lpipsPyTorch
◦ Utility components managing camera and graphics parameters
• A separate node for the Post-Training Pruning Process that takes checkpoints from training and refines the model using “prune_finetune.py” and respective scripts.
• Lastly, an Evaluation & Output node that shows rendering output, logs (train.log), results.json, and compatibility with external 3D viewers.
◦ Data flow from raw input → scene processing → training → rendering
◦ Dependencies among training modules, rendering, and loss evaluation.
◦ Feedback from the evaluation process (e.g., GPU memory logging, tensorboard integration) back to training adjustments.
Architectural Patterns & Technologies
• The system employs a modular architecture with clear separation among data handling, training, rendering, and evaluation.
• It leverages PyTorch for deep learning-related operations and GPU management.
• External submodules indicate a plug-in style to extend the functionality (e.g., differentiable mask rendering).
• Paradigm of adaptive sampling of Gaussians and mask-dependent pruning – this distinguishes the system from a standard full pipeline.