jashshah999/gtsam-splatfactors

Purpose and entry points

gtsam-splatfactors is a CUDA-oriented Python SLAM system combining 3D Gaussian Splatting with a GTSAM/iSAM2 pose graph. Camera poses are globally optimized in the factor graph while Gaussian parameters are optimized separately. Primary API and orchestration are in gsplat_slam/__init__.py, gsplat_slam/slam.py, and CLI support is in gsplat_slam/cli.py. Runnable evaluation/demo entry points are examples/eval_tum.py, examples/eval_kitti.py, and examples/synthetic_demo.py.

SLAM orchestrator and incremental graph

SplatSLAM coordinates keyframe insertion, visual odometry constraints, map updates, graph optimization, and pose retrieval. It owns the boundary between incoming RGB/RGB-D frames and persistent SLAM state. GTSAM Pose3 variables and iSAM2 receive priors, odometry, loop-closure, and optional inertial constraints; optimized poses feed later rendering and mapping stages.

Paths: gsplat_slam/slam.py, gsplat_slam/keyframe_manager.py, gsplat_slam/pose_utils.py.

Gaussian map and rendering boundary

The scene representation is a PyTorch-managed Gaussian map, rendered from candidate poses through gsplat. This GPU rendering layer is distinct from GTSAM’s CPU-side graph semantics: it supplies photometric predictions/residuals, while pose updates remain factor-graph operations. Map evolution includes initialization from depth and adaptive Gaussian densification.

Paths: gsplat_slam/map.py, gsplat_slam/renderer.py, gsplat_slam/depth_init.py, gsplat_slam/densification.py.

Photometric factor integration

GaussianSplatFactor adapts splat-rendering photometric error into a GTSAM-compatible factor. Given a pose, target image, camera intrinsics, sampled pixels, and Gaussian map, it evaluates residuals and Jacobians for graph insertion. Jacobians use GPU central differences in the appropriate SE(3)/GTSAM update convention, making rasterized image alignment usable by iSAM2.

Paths: gsplat_slam/factor.py, gsplat_slam/analytical_jacobian.py, gsplat_slam/renderer.py.

Tracking, keyframes, and mapper

Frames are converted into pose observations/keyframes, with keyframe lifecycle management separating high-frequency frame processing from more expensive mapping and graph work. The mapper alternates Gaussian optimization—via PyTorch/Adam—with graph pose corrections, keeping the learned map consistent with globally revised trajectory estimates.

Paths: gsplat_slam/mapper.py, gsplat_slam/keyframe_manager.py, gsplat_slam/slam.py.

Loop closure and sensor fusion

Loop detection proposes revisits using DINOv2 features, followed by geometric verification before adding global graph constraints. This is the key nonlocal correction path: a verified constraint is inserted into iSAM2 and can revise all affected poses. The factor graph also has an IMU-factor extension point for multi-sensor fusion; other GTSAM factor types can be added at this boundary.

Paths: gsplat_slam/loop_detector.py, gsplat_slam/imu_factor.py, gsplat_slam/factor.py.

Data ingestion, outputs, and runtime dependencies

Dataset adapters provide TUM RGB-D and KITTI inputs; video processing supports sequential runs. Export and visualization consume estimated poses/map state rather than participating in optimization. Material dependencies are Python 3.10+, PyTorch and CUDA, gsplat rasterization, OpenCV, and GTSAM/iSAM2; DINOv2 is an external learned-feature dependency for loop closure.

Paths: gsplat_slam/tum_loader.py, gsplat_slam/kitti_loader.py, gsplat_slam/video_runner.py, gsplat_slam/exporters.py.