How dali-jack/g4splat works
1. Repository shape
This is a research pipeline for 3D reconstruction, not a single standalone model. The top-level train.py is the orchestrator, with separate subsystems for scene loading, geometry estimation, generative inpainting, Gaussian optimization, rendering, mesh extraction, and evaluation.
2. Primary execution path
The main flow starts from a scene folder under data/..., then runs through geometry/structure initialization, Gaussian splatting optimization, optional generative completion of unseen regions, and finally rendering plus metric evaluation. The README notes that evaluation is integrated into train.py, so training and validation are coupled in one run.
3. Core reconstruction engine
The 2d-gaussian-splatting subtree provides the rendering and scene model backbone: scene/ defines cameras, dataset readers, and Gaussian state; gaussian_renderer/ handles the differentiable renderer; render*.py scripts produce novel views and dense outputs; extract_mesh*.py exports surfaces from the learned representation. This is the main 3D scene representation used by the project.
4. Geometry guidance layer
The planes/ and guidance/ folders implement the “geometry-guided” part of G4Splat. They contain plane extraction/refinement, inconsistency solving, chart/grid alignment, depth refinement, and utilities for integrating multi-view geometry into the reconstruction. These modules act as a bridge between raw predicted geometry and the splat/mesh representation.
5. Generative prior / unseen-region completion
The guidance/See3D_modules/ code and scripts/see3d_inpaint.py provide the generative prior used to fill or regularize unobserved regions. This is a distinct subsystem from geometric estimation: it consumes camera/view context and produces completion cues that are later merged into the scene optimization.
6. External geometry estimators
The repo vendors two major external projects:
- Depth-Anything-V2/ for monocular or metric depth estimation.
- mast3r/ for MASt3R-SfM / matching / pose estimation.
These are not just utilities; they are upstream sources of depth, matching, and camera geometry that feed the reconstruction pipeline.
7. Scene initialization and optimization variants
The configs/ directory shows multiple operating modes: mast3r, charts_alignment, adaptive_tetrahedralization, multiresolution_tsdf, and free_gaussians_refinement. That suggests the pipeline has several initialization/refinement branches depending on the input type and desired geometry representation.
8. Surface extraction and volumetric boundaries
Mesh and surface generation are handled through extract_mesh.py, extract_mesh_adaptive_tsdf.py, utils/mcube_utils.py, utils/tetmesh.py, and related helpers. The architecture supports multiple surface backends: marching cubes, TSDF-based extraction, and tetrahedralization-based processing.
9. Native acceleration boundary
Three submodules are compiled extensions and should be drawn as low-level native dependencies:
- submodules/diff-surfel-rasterization for differentiable splat/surfel rendering
- submodules/simple-knn for nearest-neighbor kernels
- submodules/tetra-triangulation for tetrahedral mesh construction
These are performance-critical C++/CUDA boundaries under the Python control plane.
10. Evaluation and metrics
The eval/ package and top-level metrics.py compute image and mesh quality. Rendering outputs are compared against ground truth after training; LPIPS is vendored under lpipsPyTorch/ for perceptual image evaluation.
11. Data and I/O conventions
Input data is expected to be preprocessed into per-scene folders, with datasets like replica, scannetpp, deepblending, and denseview. Scripts in scripts/ automate dataset-specific preprocessing, rendering, SfM runs, and refinement passes, while convert.py and view.py handle format conversion and inspection.
12. Practical system dependencies
The repo relies on PyTorch with CUDA, plus compiled geometry libraries and model checkpoints. The architecture therefore has a clear split between Python orchestration and GPU-accelerated native kernels, with pretrained external models acting as required inference services.
13. What to show in an architecture diagram
A faithful diagram should show:
- scene data in,
- MASt3R/DepthAnything geometry inputs,
- See3D generative completion as a side input,
- geometry-guided alignment/refinement in the middle,
- Gaussian splatting optimization as the main reconstruction core,
- mesh/render/eval outputs at the end,
- and the CUDA/C++ extensions as supporting infrastructure below the Python pipeline.
Main entry points: train.py train.py · gaussian_model.py Gaussian model · __init__.py Renderer · eval.py Evaluation
Components 18
Orchestrators
Entry points that wire the reconstruction run together.
- train.py · Main coupled training, validation, and evaluation driver.
train.py - Refine depth · Alternative training path that injects depth refinement.
2d-gaussian-splatting/train_with_refine_depth.py
Reconstruction core
Gaussian scene model, rendering, and extraction backbone.
- Scene I/O · Scene folders, camera parsing, and dataset readers.
2d-gaussian-splatting/scene/dataset_readers.py - Gaussian model · Stores and updates the splat representation.
2d-gaussian-splatting/scene/gaussian_model.py - Renderer · Gaussian rendering and GUI-facing render interface.
2d-gaussian-splatting/gaussian_renderer/__init__.py - Render passes · Novel-view, dense, chart, and multiresolution rendering entry points.
2d-gaussian-splatting/render.py - Mesh export · Marching cubes, adaptive TSDF, and mesh conversion utilities.
2d-gaussian-splatting/extract_mesh.py
Geometry guidance
Plane, chart, and alignment stages that refine structure.
Generative prior
See3D-based completion used for unseen regions.
- See3D · Multi-view diffusion and completion modules for unseen regions.
2d-gaussian-splatting/guidance/See3D_modules/mv_diffusion.py
External estimators
Vendored geometry sources for depth and pose.
- Depth-Anything V2 · Monocular or metric depth source used for initialization and refinement.
Depth-Anything-V2/mono_depth.py - MASt3R · Feature matching, pose estimation, and SfM-style geometry input.
mast3r/run_mast3r.py
Native kernels
Compiled acceleration boundaries under Python control.
- Rasterizer · Differentiable splat/surfel rendering kernel.
2d-gaussian-splatting/submodules/diff-surfel-rasterization - Simple KNN · Nearest-neighbor kernel used by reconstruction routines.
2d-gaussian-splatting/submodules/simple-knn - Tetra triangulation · Tetrahedral mesh construction backend.
2d-gaussian-splatting/submodules/tetra-triangulation
Outputs
Rendering, mesh export, and evaluation.
- Evaluation · Image and mesh evaluation pipeline.
2d-gaussian-splatting/eval/eval.py - Metrics · Top-level metric aggregation for reconstruction quality.
2d-gaussian-splatting/metrics.py - Outputs · Rendered views, extracted surfaces, and evaluation reports.
Connections 25
- train.py → Scene I/O: loads · Reads per-scene inputs and camera metadata.
- train.py → Depth-Anything V2: queries depth · Uses predicted depth for geometry initialization and refinement.
- train.py → MASt3R: queries poses · Consumes matching and SfM geometry when available.
- train.py → Planes: initializes structure · Builds plane-guided geometry before splat optimization.
- train.py → Guidance: aligns geometry · Runs chart alignment and inconsistency solving.
- train.py → See3D: adds prior · Optionally fills unseen regions with generative completion cues.
- train.py → Gaussian model: optimizes · Main reconstruction state updated during training.
- train.py → Renderer: renders · Uses differentiable rendering during optimization and validation.
- train.py → Evaluation: evaluates · Integrated validation and metrics run inside training.
- Refine depth → Planes: refines · Depth-aware variant tightens structural guidance.
- Scene I/O → Gaussian model: seeds · Scene content and cameras seed the Gaussian representation.
- Planes → Guidance: feeds · Plane outputs flow into alignment and inconsistency resolution.
- Guidance → Gaussian model: constrains · Guidance steers splat placement and refinement.
- See3D → Guidance: provides cues · Completion outputs are merged through guidance utilities.
- Gaussian model → Renderer: drives · Renderer consumes the learned Gaussian state.
- Renderer → Render passes: serves · Render scripts reuse the differentiable renderer.
- Renderer → Evaluation: supplies images · Rendered outputs are compared against ground truth.
- Gaussian model → Mesh export: exports · Surface extraction runs on the learned representation.
- Evaluation → Metrics: aggregates · Image and mesh metrics are computed together.
- Renderer → Rasterizer: accelerates · Native rasterizer provides the low-level rendering kernel.
- Gaussian model → Simple KNN: neighbors · Uses compiled KNN routines for geometric queries.
- Mesh export → Tetra triangulation: uses · Adaptive tetrahedral processing relies on native triangulation.
- Render passes → Outputs: writes · Produces novel views and dense render artifacts.
- Mesh export → Outputs: writes · Produces meshes and extracted surfaces.
- Evaluation → Outputs: reports · Stores evaluation summaries and quality metrics.