How szymanowiczs/splatter-image works
Main Components
• Configuration & Experiment Management:
• Folder: configs (with subdirectories for camera embeddings, datasets, experiments, Hydra, WandB)
• Purpose: Holds all hyper-parameter configurations, experiment-specific settings, and defaults.
• Data Preprocessing & Dataset Management:
• Folder: data_preprocessing (e.g. preprocess_co3d.py)
• Folder: datasets (multiple dataset reader modules for CO3D, SRN, Objaverse, etc.)
• Purpose: Downloads, preprocesses, and provides input data in various formats for different datasets.
• Model & Training:
• Files: train_network.py, scene/gaussian_predictor.py
• Purpose: The training loop (two-stage training with and without LPIPS fine-tuning) and network definition.
• Rendering:
• Folder: gaussian_renderer
• Purpose: Implements the Gaussian Splatting renderer to produce images from the 3D representation.
• Evaluation & Demo:
• Files: eval.py, gradio_app.py, demo_examples (holds demo videos)
• Purpose: Evaluation routines for multiple datasets, and an interactive demo (using Gradio) for user-upload of images and showcasing the results.
• Utilities & Testing:
• Folder: utils (general helper functions: camera, graphics, loss, etc.)
• Folder: tests (unit tests for key parts)
• Purpose: Provide supporting functionality and testing frameworks.
Relationships and Interactions
• The Configurations (configs) are used by almost every component (training, evaluation, preprocessing) to set experiment parameters and external service options (like WandB logging).
• During Data Preprocessing, raw datasets (CO3D, ShapeNet, Objaverse, GSO) are processed by scripts under data_preprocessing, and dataset modules (in datasets/) load and prepare data for training/evaluation.
• The Training Loop (in train_network.py) uses dataset loaders to feed data into the model defined in scene/gaussian_predictor.py, which in turn makes calls to the Gaussian renderer to convert the implicit 3D reconstruction into rendered images.
• The Evaluation module (eval.py) leverages the same pipeline to load a pretrained model and run validations, optionally also saving rendered visualizations using camera trajectories as implemented in the rendering and utility scripts.
• The Demo Application (gradio_app.py) provides an interactive front-end that wraps the inference/evaluation pipeline to allow users to upload images and view outputs. The demo_examples folder holds example outputs and videos, illustrating results.
• Utility modules, such as those in utils, support all layers (data handling, camera transformations, image processing, etc.), ensuring integration consistency across components.
Architectural Patterns & Principles
• Modularity: Clear separation between data handling (preprocessing and dataset loading), model training (training and evaluation), and visualization/rendering.
• Config-driven Execution: Use of YAML files in configs, as well as Hydra for flexible configuration management.
• Two-Stage Training: The training is clearly separated into a primary training stage and a fine-tuning stage using LPIPS, which can be highlighted as distinct modules or phases.
• External Integration: The system integrates with external libraries (PyTorch, PyTorch3D, Gaussian Splatting renderer, Gradio, WandB) – each of which plays a significant role.