How mit-spark/vggt-slam works
Purpose and entry points
VGGT-SLAM 2.0 performs real-time dense RGB scene reconstruction/SLAM from image sequences or a RealSense stream. Offline execution begins at main.py; live camera operation begins at main_realtime.py. Both orchestrate the vggt_slam package and can publish incremental maps to Viser.
Input and ingestion boundary
Offline mode consumes an ordered image folder; realtime mode consumes RealSense frames. Camera representation and pose/intrinsics handling are centralized in vggt_slam/cameras.py. The ingestion boundary feeds RGB observations into the mapping pipeline rather than exposing a web or service API.
Feed-forward reconstruction and local submaps
Frames are grouped into bounded local reconstruction units, implemented by vggt_slam/submap.py. A submap is the principal unit for dense geometry, camera estimates, map updates, and configurable update frequency; reducing submap size gives more frequent live updates. Image-pair/frame overlap logic in vggt_slam/frame_overlap.py determines useful associations within or across these local units.
Global map and graph state
The persistent reconstruction is managed in vggt_slam/map.py, while inter-submap/camera relations are represented by vggt_slam/graph.py. This separates dense local outputs from globally optimized state: new submaps add observations and constraints, then corrected global poses propagate to map placement.
Loop closure and optimization
Loop candidates and closure constraints are handled in vggt_slam/loop_closure.py. Global geometric optimization is implemented in vggt_slam/solver.py, with scale-specific estimation in vggt_slam/scale_solver.py. The control flow is: build local submaps → detect limited loop closures → add graph constraints → optimize global poses/scales → update the assembled map.
Visualization and optional semantic detection
vggt_slam/viewer.py publishes cameras, frustums, images, point clouds, and optional 3D detections to Viser. Visualization is downstream of mapping and can voxel-sparsify only transmitted display points. Optional open-set 3D object detection is enabled from the entry points and relies on third-party Perception Encoder and SAM 3 dependencies installed by setup.sh; it is not required for SLAM.
Runtime and external dependencies
The runtime is Python 3.11 with dependencies declared in requirements.txt and installation/bootstrap logic in setup.sh. Core reconstruction depends on the project’s VGGT fork; optimization incorporates SL(4)-oriented graph solving and likely GTSAM support. External operational dependencies include RealSense hardware for main_realtime.py, Viser for visualization, and ffmpeg only for converting video captures into offline image sequences.
Main entry points: submap.py Local submaps · graph.py Pose graph · solver.py Global solver · main.py Offline runner
Components 16
Entry & Inputs
- Offline runner · Processes ordered image folders.
main.py - Realtime runner · Processes live RealSense frames.
main_realtime.py - Image sequence
- RealSense camera
SLAM Core
- Camera handling · Centralizes poses and intrinsics.
vggt_slam/cameras.py - Local submaps · Bounded units for geometry, cameras, and map updates.
vggt_slam/submap.py - Frame overlap · Selects useful within- and cross-submap associations.
vggt_slam/frame_overlap.py - VGGT fork · Provides core feed-forward reconstruction.
Global Consistency
- Persistent map · Assembles local reconstruction outputs.
vggt_slam/map.py - Pose graph · Stores camera and inter-submap relations.
vggt_slam/graph.py - Loop closure · Finds limited loop candidates and closures.
vggt_slam/loop_closure.py - Global solver · Optimizes graph geometry and global poses.
vggt_slam/solver.py - Scale solver
vggt_slam/scale_solver.py
Outputs & Optional Features
- Viser publisher · Publishes incremental cameras, imagery, and point clouds.
vggt_slam/viewer.py - Open-set 3D detection · Uses Perception Encoder and SAM 3.
setup.sh - Python runtime · Python 3.11 dependencies and bootstrap installation.
requirements.txt
Connections 21
- Image sequence → Offline runner: input
- RealSense camera → Realtime runner: frames
- Offline runner → Camera handling: orchestrates
- Realtime runner → Camera handling: orchestrates
- Camera handling → Local submaps: RGB observations
- Frame overlap → Local submaps: associations
- Local submaps → VGGT fork: reconstructs with
- VGGT fork → Local submaps: dense local outputs
- Local submaps → Persistent map: observations and geometry
- Local submaps → Pose graph: poses and constraints
- Frame overlap → Loop closure: cross-unit matches
- Pose graph → Loop closure: search context
- Loop closure → Pose graph: closure constraints
- Pose graph → Global solver: graph state
- Scale solver → Global solver: scale estimates
- Global solver → Pose graph: corrected poses
- Global solver → Persistent map: corrected placement
- Persistent map → Viser publisher: incremental map
- Open-set 3D detection → Viser publisher: optional 3D detections
- Python runtime → Offline runner: runs
- Python runtime → Realtime runner: runs