How yjb6/igs works
Project type and purpose
• This is a research-oriented, Python-based computer vision package for dynamic scene reconstruction via “Instant Gaussian Stream.” It provides two primary pipelines:
• Streaming reconstruction (initial Gaussian training with RaDe-GS, compression, per-frame streaming inference)
• AGM-Net training (learning to refine Gaussian streams)
• It is not a web or full-stack app, but rather a modular ML system leveraging PyTorch, custom CUDA rasterizers, and external submodules.
Top-level structure and architectural choices
• Top-level directories:
• igs/ → core library: data loaders, model definitions, utilities
• configs/ → YAML configurations for demo, training, evaluation
• acc_cfg/ → accelerate config for distributed training
• submodules/ → external code (RaDe-GS and its CUDA rasterizers, tetra triangulation, etc.)
• script/ → data-preparation and helper scripts (subsampling, pairing, video conversion)
• lpipsPyTorch/ → perceptual metric module
• infer_batch.py & main.py → CLI entry points for inference and training
• Architecture pattern:
• Modular python package (igs) separating data, models, and utils
• Config-driven execution (YAML files consumed by main.py and infer_batch.py)
• External submodules integrated as pip-installable packages (CUDA rasterizers, torch_cluster)
Core components and interactions
A. Data Ingestion & Preparation
• Raw multi-view images + Colmap outputs → script/pre_test_data.sh, subsample.py → structured on disk
• build_3dgs_dataset.py for large-scale dataset construction
B. Configuration
• configs/*.yaml define model hyperparameters, data paths, training/eval settings
• acc_cfg/default_config.yaml controls distributed setup
C. Core Library (igs/)
• igs.data: Data loaders (data.py, infer_data.py), utilities for batching and preprocessing
• igs.models:
– gaussian_model.py & grid_encoder.py: Core Gaussian Splatting data structures
– networks.py, transformers.py, unet.py: Neural refinement architectures (AGM-Net)
– unimatch/: Transformer-based matching submodule for temporal coherence
• igs.utils: Config parsing, image/graphics utils, loss computation, I/O and system utilities
D. Execution Entry Points
• main.py → launches training of AGM-Net via accelerate
• infer_batch.py → streaming inference on key-candidate sequences, renders per-frame outputs
E. External Libraries & Submodules
• RaDe-GS (submodules/RaDe-GS): Initial Gaussian training & compression pipelines
• diff-gaussian-rasterization(::clamp) CUDA rasterizers for splatting
• tetra_triangulation: Auxiliary C++ triangulation Python bindings
• lpipsPyTorch: Perceptual loss metric
• torch_cluster, PyTorch 2.0+, torchvision, torchaudio