# jashshah999/gtsam-splatfactors architecture > Architecture diagram of the GitHub repository jashshah999/gtsam-splatfactors, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-04). It is an AI-made overview: check details against the source. - Interactive diagram: https://gitdiagram.com/jashshah999/gtsam-splatfactors - Repository: https://github.com/jashshah999/gtsam-splatfactors ## Overview #### Purpose and entry points `gtsam-splatfactors` is a CUDA-oriented Python SLAM system combining 3D Gaussian Splatting with a GTSAM/iSAM2 pose graph. Camera poses are globally optimized in the factor graph while Gaussian parameters are optimized separately. Primary API and orchestration are in `gsplat_slam/__init__.py`, `gsplat_slam/slam.py`, and CLI support is in `gsplat_slam/cli.py`. Runnable evaluation/demo entry points are `examples/eval_tum.py`, `examples/eval_kitti.py`, and `examples/synthetic_demo.py`. #### SLAM orchestrator and incremental graph `SplatSLAM` coordinates keyframe insertion, visual odometry constraints, map updates, graph optimization, and pose retrieval. It owns the boundary between incoming RGB/RGB-D frames and persistent SLAM state. GTSAM Pose3 variables and iSAM2 receive priors, odometry, loop-closure, and optional inertial constraints; optimized poses feed later rendering and mapping stages. Paths: `gsplat_slam/slam.py`, `gsplat_slam/keyframe_manager.py`, `gsplat_slam/pose_utils.py`. #### Gaussian map and rendering boundary The scene representation is a PyTorch-managed Gaussian map, rendered from candidate poses through `gsplat`. This GPU rendering layer is distinct from GTSAM’s CPU-side graph semantics: it supplies photometric predictions/residuals, while pose updates remain factor-graph operations. Map evolution includes initialization from depth and adaptive Gaussian densification. Paths: `gsplat_slam/map.py`, `gsplat_slam/renderer.py`, `gsplat_slam/depth_init.py`, `gsplat_slam/densification.py`. #### Photometric factor integration `GaussianSplatFactor` adapts splat-rendering photometric error into a GTSAM-compatible factor. Given a pose, target image, camera intrinsics, sampled pixels, and Gaussian map, it evaluates residuals and Jacobians for graph insertion. Jacobians use GPU central differences in the appropriate SE(3)/GTSAM update convention, making rasterized image alignment usable by iSAM2. Paths: `gsplat_slam/factor.py`, `gsplat_slam/analytical_jacobian.py`, `gsplat_slam/renderer.py`. #### Tracking, keyframes, and mapper Frames are converted into pose observations/keyframes, with keyframe lifecycle management separating high-frequency frame processing from more expensive mapping and graph work. The mapper alternates Gaussian optimization—via PyTorch/Adam—with graph pose corrections, keeping the learned map consistent with globally revised trajectory estimates. Paths: `gsplat_slam/mapper.py`, `gsplat_slam/keyframe_manager.py`, `gsplat_slam/slam.py`. #### Loop closure and sensor fusion Loop detection proposes revisits using DINOv2 features, followed by geometric verification before adding global graph constraints. This is the key nonlocal correction path: a verified constraint is inserted into iSAM2 and can revise all affected poses. The factor graph also has an IMU-factor extension point for multi-sensor fusion; other GTSAM factor types can be added at this boundary. Paths: `gsplat_slam/loop_detector.py`, `gsplat_slam/imu_factor.py`, `gsplat_slam/factor.py`. #### Data ingestion, outputs, and runtime dependencies Dataset adapters provide TUM RGB-D and KITTI inputs; video processing supports sequential runs. Export and visualization consume estimated poses/map state rather than participating in optimization. Material dependencies are Python 3.10+, PyTorch and CUDA, `gsplat` rasterization, OpenCV, and GTSAM/iSAM2; DINOv2 is an external learned-feature dependency for loop closure. Paths: `gsplat_slam/tum_loader.py`, `gsplat_slam/kitti_loader.py`, `gsplat_slam/video_runner.py`, `gsplat_slam/exporters.py`. Main entry points: [`gsplat_slam/slam.py`](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/slam.py) (SplatSLAM), [`gsplat_slam/map.py`](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/map.py) (Gaussian map), [`gsplat_slam/factor.py`](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/factor.py) (GaussianSplatFactor), [`gsplat_slam/mapper.py`](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/mapper.py) (Mapper) ## Diagram (Mermaid) ```mermaid flowchart TD subgraph group_ingest["Input & Runs"] node_api["Python API
package entry
[__init__.py]"] node_cli["CLI
command interface
[cli.py]"] node_datasets["TUM/KITTI adapters
dataset ingestion
[tum_loader.py]"] node_video["Video runner
sequence input
[video_runner.py]"] end subgraph group_slam["SLAM & Graph"] node_slam_orchestrator["SplatSLAM
SLAM orchestrator
[slam.py]"] node_keyframes["Keyframe manager
keyframe lifecycle"] node_pose_utils["Pose utilities
Pose3 conversion
[pose_utils.py]"] node_isam2[("GTSAM iSAM2 graph
incremental pose graph")] node_photometric_factor{{"GaussianSplatFactor
GTSAM photometric factor
[factor.py]"}} end subgraph group_mapping["Gaussian Mapping"] node_gaussian_map[("Gaussian map
PyTorch scene map
[map.py]")] node_depth_init["Depth initialization
map initialization
[depth_init.py]"] node_renderer["gsplat renderer
GPU rasterization
[renderer.py]"] node_jacobians["GPU Jacobians
central-difference SE(3)"] node_mapper["Mapper
map optimization
[mapper.py]"] node_densification["Adaptive densification
map refinement
[densification.py]"] end subgraph group_closure["Global Correction"] node_loop_detector["Loop detector
DINOv2 loop closure
[loop_detector.py]"] node_imu{{"IMU factor extension
sensor-fusion factor
[imu_factor.py]"}} end subgraph group_output["Outputs"] node_exporters["Export & visualization
result consumer
[exporters.py]"] end node_api -->|"creates"| node_slam_orchestrator node_cli -->|"runs"| node_slam_orchestrator node_datasets -->|"RGB/RGB-D frames"| node_slam_orchestrator node_video -->|"frame sequence"| node_slam_orchestrator node_slam_orchestrator -->|"inserts frames"| node_keyframes node_slam_orchestrator -->|"updates graph"| node_isam2 node_pose_utils -->|"Pose3 handling"| node_isam2 node_keyframes -->|"mapping work"| node_mapper node_depth_init -->|"initializes"| node_gaussian_map node_mapper -->|"Adam updates"| node_gaussian_map node_densification -->|"refines"| node_gaussian_map node_gaussian_map -->|"scene parameters"| node_renderer node_renderer -->|"photometric residual"| node_photometric_factor node_jacobians -->|"residual Jacobians"| node_photometric_factor node_photometric_factor -->|"pose constraint"| node_isam2 node_isam2 -->|"optimized poses"| node_mapper node_loop_detector -->|"verified loop constraint"| node_isam2 node_imu -.->|"inertial constraint"| node_isam2 node_isam2 -->|"estimated poses"| node_exporters node_gaussian_map -->|"map state"| node_exporters click node_api "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/__init__.py" click node_cli "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/cli.py" click node_datasets "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/tum_loader.py" click node_video "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/video_runner.py" click node_slam_orchestrator "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/slam.py" click node_keyframes "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/keyframe_manager.py" click node_pose_utils "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/pose_utils.py" click node_photometric_factor "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/factor.py" click node_gaussian_map "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/map.py" click node_depth_init "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/depth_init.py" click node_renderer "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/renderer.py" click node_jacobians "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/analytical_jacobian.py" click node_mapper "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/mapper.py" click node_densification "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/densification.py" click node_loop_detector "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/loop_detector.py" click node_imu "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/imu_factor.py" click node_exporters "https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/exporters.py" classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554 classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337 classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81 classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a class node_api,node_cli,node_datasets,node_video toneBlue class node_slam_orchestrator,node_keyframes,node_pose_utils,node_isam2,node_photometric_factor toneAmber class node_gaussian_map,node_depth_init,node_renderer,node_jacobians,node_mapper,node_densification toneMint class node_loop_detector,node_imu toneRose class node_exporters toneIndigo ``` ## Components ### Input & Runs - [Python API](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/__init__.py): package entry · Primary library orchestration surface. · `gsplat_slam/__init__.py` - [CLI](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/cli.py): command interface · `gsplat_slam/cli.py` - [TUM/KITTI adapters](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/tum_loader.py): dataset ingestion · `gsplat_slam/tum_loader.py` - [Video runner](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/video_runner.py): sequence input · `gsplat_slam/video_runner.py` ### SLAM & Graph - [SplatSLAM](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/slam.py): SLAM orchestrator · Owns frame-to-persistent-SLAM-state processing. · `gsplat_slam/slam.py` - [Keyframe manager](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/keyframe_manager.py): keyframe lifecycle · Separates frequent frame handling from costly map and graph work. · `gsplat_slam/keyframe_manager.py` - [Pose utilities](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/pose_utils.py): Pose3 conversion · `gsplat_slam/pose_utils.py` - **GTSAM iSAM2 graph**: incremental pose graph · Optimizes Pose3 variables and global constraints. - [GaussianSplatFactor](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/factor.py): GTSAM photometric factor · Bridges rendered image residuals into graph optimization. · `gsplat_slam/factor.py` ### Gaussian Mapping - [Gaussian map](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/map.py): PyTorch scene map · Persistent learnable 3D Gaussian representation. · `gsplat_slam/map.py` - [Depth initialization](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/depth_init.py): map initialization · `gsplat_slam/depth_init.py` - [gsplat renderer](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/renderer.py): GPU rasterization · Renders candidate poses for photometric alignment. · `gsplat_slam/renderer.py` - [GPU Jacobians](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/analytical_jacobian.py): central-difference SE(3) · Computes renderer residual Jacobians in GTSAM update convention. · `gsplat_slam/analytical_jacobian.py` - [Mapper](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/mapper.py): map optimization · Alternates Adam Gaussian updates with graph pose corrections. · `gsplat_slam/mapper.py` - [Adaptive densification](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/densification.py): map refinement · `gsplat_slam/densification.py` ### Global Correction - [Loop detector](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/loop_detector.py): DINOv2 loop closure · Proposes revisits and geometrically verifies them. · `gsplat_slam/loop_detector.py` - [IMU factor extension](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/imu_factor.py): sensor-fusion factor · `gsplat_slam/imu_factor.py` ### Outputs - [Export & visualization](https://github.com/jashshah999/gtsam-splatfactors/blob/main/gsplat_slam/exporters.py): result consumer · `gsplat_slam/exporters.py` ## Connections - Python API → SplatSLAM: creates - CLI → SplatSLAM: runs - TUM/KITTI adapters → SplatSLAM: RGB/RGB-D frames - Video runner → SplatSLAM: frame sequence - SplatSLAM → Keyframe manager: inserts frames - SplatSLAM → GTSAM iSAM2 graph: updates graph - Pose utilities → GTSAM iSAM2 graph: Pose3 handling - Keyframe manager → Mapper: mapping work - Depth initialization → Gaussian map: initializes - Mapper → Gaussian map: Adam updates - Adaptive densification → Gaussian map: refines - Gaussian map → gsplat renderer: scene parameters - gsplat renderer → GaussianSplatFactor: photometric residual - GPU Jacobians → GaussianSplatFactor: residual Jacobians - GaussianSplatFactor → GTSAM iSAM2 graph: pose constraint - GTSAM iSAM2 graph → Mapper: optimized poses - Loop detector → GTSAM iSAM2 graph: verified loop constraint - IMU factor extension → GTSAM iSAM2 graph: inertial constraint - GTSAM iSAM2 graph → Export & visualization: estimated poses - Gaussian map → Export & visualization: map state