# dali-jack/g4splat architecture

> Architecture diagram of the GitHub repository dali-jack/g4splat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dali-jack/g4splat
- Repository: https://github.com/dali-jack/g4splat

## Overview

1. **Repository shape**
   This is a research pipeline for 3D reconstruction, not a single standalone model. The top-level `train.py` is the orchestrator, with separate subsystems for scene loading, geometry estimation, generative inpainting, Gaussian optimization, rendering, mesh extraction, and evaluation.

2. **Primary execution path**
   The main flow starts from a scene folder under `data/...`, then runs through geometry/structure initialization, Gaussian splatting optimization, optional generative completion of unseen regions, and finally rendering plus metric evaluation. The README notes that evaluation is integrated into `train.py`, so training and validation are coupled in one run.

3. **Core reconstruction engine**
   The `2d-gaussian-splatting` subtree provides the rendering and scene model backbone: `scene/` defines cameras, dataset readers, and Gaussian state; `gaussian_renderer/` handles the differentiable renderer; `render*.py` scripts produce novel views and dense outputs; `extract_mesh*.py` exports surfaces from the learned representation. This is the main 3D scene representation used by the project.

4. **Geometry guidance layer**
   The `planes/` and `guidance/` folders implement the “geometry-guided” part of G4Splat. They contain plane extraction/refinement, inconsistency solving, chart/grid alignment, depth refinement, and utilities for integrating multi-view geometry into the reconstruction. These modules act as a bridge between raw predicted geometry and the splat/mesh representation.

5. **Generative prior / unseen-region completion**
   The `guidance/See3D_modules/` code and `scripts/see3d_inpaint.py` provide the generative prior used to fill or regularize unobserved regions. This is a distinct subsystem from geometric estimation: it consumes camera/view context and produces completion cues that are later merged into the scene optimization.

6. **External geometry estimators**
   The repo vendors two major external projects:
   - `Depth-Anything-V2/` for monocular or metric depth estimation.
   - `mast3r/` for MASt3R-SfM / matching / pose estimation.
   These are not just utilities; they are upstream sources of depth, matching, and camera geometry that feed the reconstruction pipeline.

7. **Scene initialization and optimization variants**
   The `configs/` directory shows multiple operating modes: `mast3r`, `charts_alignment`, `adaptive_tetrahedralization`, `multiresolution_tsdf`, and `free_gaussians_refinement`. That suggests the pipeline has several initialization/refinement branches depending on the input type and desired geometry representation.

8. **Surface extraction and volumetric boundaries**
   Mesh and surface generation are handled through `extract_mesh.py`, `extract_mesh_adaptive_tsdf.py`, `utils/mcube_utils.py`, `utils/tetmesh.py`, and related helpers. The architecture supports multiple surface backends: marching cubes, TSDF-based extraction, and tetrahedralization-based processing.

9. **Native acceleration boundary**
   Three submodules are compiled extensions and should be drawn as low-level native dependencies:
   - `submodules/diff-surfel-rasterization` for differentiable splat/surfel rendering
   - `submodules/simple-knn` for nearest-neighbor kernels
   - `submodules/tetra-triangulation` for tetrahedral mesh construction
   These are performance-critical C++/CUDA boundaries under the Python control plane.

10. **Evaluation and metrics**
   The `eval/` package and top-level `metrics.py` compute image and mesh quality. Rendering outputs are compared against ground truth after training; LPIPS is vendored under `lpipsPyTorch/` for perceptual image evaluation.

11. **Data and I/O conventions**
   Input data is expected to be preprocessed into per-scene folders, with datasets like `replica`, `scannetpp`, `deepblending`, and `denseview`. Scripts in `scripts/` automate dataset-specific preprocessing, rendering, SfM runs, and refinement passes, while `convert.py` and `view.py` handle format conversion and inspection.

12. **Practical system dependencies**
   The repo relies on PyTorch with CUDA, plus compiled geometry libraries and model checkpoints. The architecture therefore has a clear split between Python orchestration and GPU-accelerated native kernels, with pretrained external models acting as required inference services.

13. **What to show in an architecture diagram**
   A faithful diagram should show:
   - scene data in,
   - MASt3R/DepthAnything geometry inputs,
   - See3D generative completion as a side input,
   - geometry-guided alignment/refinement in the middle,
   - Gaussian splatting optimization as the main reconstruction core,
   - mesh/render/eval outputs at the end,
   - and the CUDA/C++ extensions as supporting infrastructure below the Python pipeline.

Main entry points: [`train.py`](https://github.com/dali-jack/g4splat/blob/main/train.py) (train.py), [`2d-gaussian-splatting/scene/gaussian_model.py`](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/scene/gaussian_model.py) (Gaussian model), [`2d-gaussian-splatting/gaussian_renderer/__init__.py`](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/gaussian_renderer/__init__.py) (Renderer), [`2d-gaussian-splatting/eval/eval.py`](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/eval/eval.py) (Evaluation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_root["Orchestrators"]
  node_train_py["train.py<br/>orchestrator<br/>[train.py]"]
  node_refine_depth_py["Refine depth<br/>aux pipeline"]
end

subgraph group_grp_core["Reconstruction core"]
  node_scene_readers["Scene I/O<br/>dataset loading<br/>[dataset_readers.py]"]
  node_gaussian_model[("Gaussian model<br/>scene state<br/>[gaussian_model.py]")]
  node_renderer_core["Renderer<br/>differentiable render<br/>[__init__.py]"]
  node_render_scripts["Render passes<br/>render scripts<br/>[render.py]"]
  node_mesh_extractors["Mesh export<br/>surface extraction<br/>[extract_mesh.py]"]
end

subgraph group_grp_geom["Geometry guidance"]
  node_planes_layer["Planes<br/>geometry init<br/>[plane_excavator.py]"]
  node_guidance_layer["Guidance<br/>geometry bridge<br/>[see3d_util.py]"]
end

subgraph group_grp_prior["Generative prior"]
  node_see3d_modules["See3D<br/>generative prior<br/>[mv_diffusion.py]"]
end

subgraph group_grp_ext["External estimators"]
  node_depth_anything["Depth-Anything V2<br/>depth estimator<br/>[mono_depth.py]"]
  node_mast3r["MASt3R<br/>matching/sfm<br/>[run_mast3r.py]"]
end

subgraph group_grp_native["Native kernels"]
  node_diff_raster{{"Rasterizer<br/>CUDA extension"}}
  node_simple_knn{{"Simple KNN<br/>CUDA extension"}}
  node_tetra_triang{{"Tetra triangulation<br/>native geometry"}}
end

subgraph group_grp_eval["Outputs"]
  node_eval_pkg["Evaluation<br/>metrics<br/>[eval.py]"]
  node_metrics_py["Metrics<br/>[metrics.py]"]
  node_outputs(("Outputs<br/>renders+meshes"))
end

node_train_py -->|"loads"| node_scene_readers
node_train_py -->|"queries depth"| node_depth_anything
node_train_py -->|"queries poses"| node_mast3r
node_train_py -->|"initializes structure"| node_planes_layer
node_train_py -->|"aligns geometry"| node_guidance_layer
node_train_py -->|"adds prior"| node_see3d_modules
node_train_py -->|"optimizes"| node_gaussian_model
node_train_py -->|"renders"| node_renderer_core
node_train_py -->|"evaluates"| node_eval_pkg
node_refine_depth_py -->|"refines"| node_planes_layer
node_scene_readers -->|"seeds"| node_gaussian_model
node_planes_layer -->|"feeds"| node_guidance_layer
node_guidance_layer -->|"constrains"| node_gaussian_model
node_see3d_modules -->|"provides cues"| node_guidance_layer
node_gaussian_model -->|"drives"| node_renderer_core
node_renderer_core -->|"serves"| node_render_scripts
node_renderer_core -->|"supplies images"| node_eval_pkg
node_gaussian_model -->|"exports"| node_mesh_extractors
node_eval_pkg -->|"aggregates"| node_metrics_py
node_renderer_core -->|"accelerates"| node_diff_raster
node_gaussian_model -->|"neighbors"| node_simple_knn
node_mesh_extractors -->|"uses"| node_tetra_triang
node_render_scripts -->|"writes"| node_outputs
node_mesh_extractors -->|"writes"| node_outputs
node_eval_pkg -->|"reports"| node_outputs

click node_train_py "https://github.com/dali-jack/g4splat/blob/main/train.py"
click node_refine_depth_py "https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/train_with_refine_depth.py"
click node_scene_readers "https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/scene/dataset_readers.py"
click node_gaussian_model "https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/scene/gaussian_model.py"
click node_renderer_core "https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/gaussian_renderer/__init__.py"
click node_render_scripts "https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/render.py"
click node_mesh_extractors "https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/extract_mesh.py"
click node_planes_layer "https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/planes/plane_excavator.py"
click node_guidance_layer "https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/guidance/see3d_util.py"
click node_see3d_modules "https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/guidance/See3D_modules/mv_diffusion.py"
click node_depth_anything "https://github.com/dali-jack/g4splat/blob/main/Depth-Anything-V2/mono_depth.py"
click node_mast3r "https://github.com/dali-jack/g4splat/blob/main/mast3r/run_mast3r.py"
click node_eval_pkg "https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/eval/eval.py"
click node_metrics_py "https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/metrics.py"
click node_diff_raster "https://github.com/dali-jack/g4splat/tree/main/2d-gaussian-splatting/submodules/diff-surfel-rasterization"
click node_simple_knn "https://github.com/dali-jack/g4splat/tree/main/2d-gaussian-splatting/submodules/simple-knn"
click node_tetra_triang "https://github.com/dali-jack/g4splat/tree/main/2d-gaussian-splatting/submodules/tetra-triangulation"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_train_py,node_refine_depth_py,node_eval_pkg,node_metrics_py,node_outputs toneBlue
class node_scene_readers,node_gaussian_model,node_renderer_core,node_render_scripts,node_mesh_extractors toneAmber
class node_planes_layer,node_guidance_layer toneMint
class node_see3d_modules toneRose
class node_depth_anything,node_mast3r toneIndigo
class node_diff_raster,node_simple_knn,node_tetra_triang toneTeal
```

## Components

### Orchestrators

Entry points that wire the reconstruction run together.

- [train.py](https://github.com/dali-jack/g4splat/blob/main/train.py): orchestrator · Main coupled training, validation, and evaluation driver. · `train.py`
- [Refine depth](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/train_with_refine_depth.py): aux pipeline · Alternative training path that injects depth refinement. · `2d-gaussian-splatting/train_with_refine_depth.py`

### Reconstruction core

Gaussian scene model, rendering, and extraction backbone.

- [Scene I/O](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/scene/dataset_readers.py): dataset loading · Scene folders, camera parsing, and dataset readers. · `2d-gaussian-splatting/scene/dataset_readers.py`
- [Gaussian model](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/scene/gaussian_model.py): scene state · Stores and updates the splat representation. · `2d-gaussian-splatting/scene/gaussian_model.py`
- [Renderer](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/gaussian_renderer/__init__.py): differentiable render · Gaussian rendering and GUI-facing render interface. · `2d-gaussian-splatting/gaussian_renderer/__init__.py`
- [Render passes](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/render.py): render scripts · Novel-view, dense, chart, and multiresolution rendering entry points. · `2d-gaussian-splatting/render.py`
- [Mesh export](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/extract_mesh.py): surface extraction · Marching cubes, adaptive TSDF, and mesh conversion utilities. · `2d-gaussian-splatting/extract_mesh.py`

### Geometry guidance

Plane, chart, and alignment stages that refine structure.

- [Planes](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/planes/plane_excavator.py): geometry init · Plane extraction, depth refinement, and global plane assembly. · `2d-gaussian-splatting/planes/plane_excavator.py`
- [Guidance](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/guidance/see3d_util.py): geometry bridge · Alignment, inconsistency solving, merging, and camera helpers. · `2d-gaussian-splatting/guidance/see3d_util.py`

### Generative prior

See3D-based completion used for unseen regions.

- [See3D](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/guidance/See3D_modules/mv_diffusion.py): generative prior · Multi-view diffusion and completion modules for unseen regions. · `2d-gaussian-splatting/guidance/See3D_modules/mv_diffusion.py`

### External estimators

Vendored geometry sources for depth and pose.

- [Depth-Anything V2](https://github.com/dali-jack/g4splat/blob/main/Depth-Anything-V2/mono_depth.py): depth estimator · Monocular or metric depth source used for initialization and refinement. · `Depth-Anything-V2/mono_depth.py`
- [MASt3R](https://github.com/dali-jack/g4splat/blob/main/mast3r/run_mast3r.py): matching/sfm · Feature matching, pose estimation, and SfM-style geometry input. · `mast3r/run_mast3r.py`

### Native kernels

Compiled acceleration boundaries under Python control.

- [Rasterizer](https://github.com/dali-jack/g4splat/tree/main/2d-gaussian-splatting/submodules/diff-surfel-rasterization): CUDA extension · Differentiable splat/surfel rendering kernel. · `2d-gaussian-splatting/submodules/diff-surfel-rasterization`
- [Simple KNN](https://github.com/dali-jack/g4splat/tree/main/2d-gaussian-splatting/submodules/simple-knn): CUDA extension · Nearest-neighbor kernel used by reconstruction routines. · `2d-gaussian-splatting/submodules/simple-knn`
- [Tetra triangulation](https://github.com/dali-jack/g4splat/tree/main/2d-gaussian-splatting/submodules/tetra-triangulation): native geometry · Tetrahedral mesh construction backend. · `2d-gaussian-splatting/submodules/tetra-triangulation`

### Outputs

Rendering, mesh export, and evaluation.

- [Evaluation](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/eval/eval.py): metrics · Image and mesh evaluation pipeline. · `2d-gaussian-splatting/eval/eval.py`
- [Metrics](https://github.com/dali-jack/g4splat/blob/main/2d-gaussian-splatting/metrics.py): quality metrics · Top-level metric aggregation for reconstruction quality. · `2d-gaussian-splatting/metrics.py`
- **Outputs**: renders+meshes · Rendered views, extracted surfaces, and evaluation reports.

## Connections

- train.py → Scene I/O: loads · Reads per-scene inputs and camera metadata.
- train.py → Depth-Anything V2: queries depth · Uses predicted depth for geometry initialization and refinement.
- train.py → MASt3R: queries poses · Consumes matching and SfM geometry when available.
- train.py → Planes: initializes structure · Builds plane-guided geometry before splat optimization.
- train.py → Guidance: aligns geometry · Runs chart alignment and inconsistency solving.
- train.py → See3D: adds prior · Optionally fills unseen regions with generative completion cues.
- train.py → Gaussian model: optimizes · Main reconstruction state updated during training.
- train.py → Renderer: renders · Uses differentiable rendering during optimization and validation.
- train.py → Evaluation: evaluates · Integrated validation and metrics run inside training.
- Refine depth → Planes: refines · Depth-aware variant tightens structural guidance.
- Scene I/O → Gaussian model: seeds · Scene content and cameras seed the Gaussian representation.
- Planes → Guidance: feeds · Plane outputs flow into alignment and inconsistency resolution.
- Guidance → Gaussian model: constrains · Guidance steers splat placement and refinement.
- See3D → Guidance: provides cues · Completion outputs are merged through guidance utilities.
- Gaussian model → Renderer: drives · Renderer consumes the learned Gaussian state.
- Renderer → Render passes: serves · Render scripts reuse the differentiable renderer.
- Renderer → Evaluation: supplies images · Rendered outputs are compared against ground truth.
- Gaussian model → Mesh export: exports · Surface extraction runs on the learned representation.
- Evaluation → Metrics: aggregates · Image and mesh metrics are computed together.
- Renderer → Rasterizer: accelerates · Native rasterizer provides the low-level rendering kernel.
- Gaussian model → Simple KNN: neighbors · Uses compiled KNN routines for geometric queries.
- Mesh export → Tetra triangulation: uses · Adaptive tetrahedral processing relies on native triangulation.
- Render passes → Outputs: writes · Produces novel views and dense render artifacts.
- Mesh export → Outputs: writes · Produces meshes and extracted surfaces.
- Evaluation → Outputs: reports · Stores evaluation summaries and quality metrics.
