How lkeab/gaussian-grouping works
Major Components
• Data Input and Preprocessing:
– Components found in folders such as “deva/dataset” (for static datasets, VOS/VPS datasets, etc.), “config/…” and “utils” (e.g. image utilities) serve for data loading, preprocessing, and configuration.
– External cues: the system leverages 2D mask predictions from SAM to supervise the identity encoding. Show a connection from a “Segment Anything” or SAM module that provides 2D masks.
• Core 3D Reconstruction Engine:
– The backbone for 3D scene synthesis comes from the extended Gaussian Splatting implementation. This is implemented in modules like “deva/model”, “deva/inference” and portions of the “Tracking-Anything-with-DEVA” folder.
– Include subcomponents for geometry handling and rendering (possibly also linking to the CUDA-accelerated “diff-gaussian-rasterization” submodule).
• Segmentation and Identity Encoding Module:
– Augmented Gaussians hold compact identity encodings that enable grouping into instances or “stuff”. This logic is spread throughout the “deva” submodules (model, ext, inference) and utilities for consistency.
• Differentiable Rendering Loop and Training Pipeline:
– The training routines and loss calculations are implemented in files such as “deva/train.py”, “deva/model/trainer.py”, and their associated utility files.
– Emphasize the loop: input images + SAM masks are used to supervise the rendered output (using differentiable rendering) so that the 3D Gaussians learn both appearance and grouping.
• Scene Editing and Post-processing Module:
– This module takes the grouped Gaussians and applies local editing—for example, object removal, inpainting, object style transfer, and scene recomposition.
– The “docs” and “script” folders contain demonstration scripts and editing utilities (e.g. “edit_object_inpaint.py”, “edit_object_removal.py”) which show how to separate a group and modify it.
• Evaluation and Utilities:
– Modules for evaluation (found in “evaluation” and “deva/vps_metrics”), dataset preparation (in “scripts” or “config” folders) and auxiliary analysis (e.g., “utils”, “scene”) are also important.
– These components validate the segmentation quality (e.g., LERF-Mask dataset evaluation) and support experiment reproducibility.
• External Dependencies and Accelerated Submodules:
Relationships and Data Flow
• Input flows start from multi-view images and corresponding 2D masks (via SAM) that are fed into the Data Input module.
• The reconstruction engine uses these inputs to initialize and adjust 3D Gaussians (with geometric and appearance attributes).
• A clear loop should be depicted where the identity encodings are adjusted by comparing the rendered outputs with the supervisory 2D masks.
• The grouped Gaussians pass to the scene editing module where various 3D operations (removal, inpainting, style transfer) can be applied interactively or automatically.
• Downstream, evaluation modules analyze the results using metrics computed in “deva/vps_metrics” or via external evaluation scripts.
Architectural Patterns
• Layered Architecture: Different layers exist—from raw data ingestion, to core 3D modeling, to segmentation and editing, and finally to evaluation.
• Modular Design: Each submodule (e.g. “deva”, “lama”, “ext”, “scripts”) is responsible for an aspect of the pipeline—promote reuse and experimentation.
• Integration of CUDA/Low-Level Acceleration: Note that certain submodules (“diff-gaussian-rasterization”, “simple-knn”) are integrated to accelerate numerical computations.
• Differentiable Rendering: This forms a key feedback loop in the training phase, where the 3D scene representation and segmentation cues are jointly refined.