# mit-spark/vggt-slam architecture

> Architecture diagram of the GitHub repository mit-spark/vggt-slam, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mit-spark/vggt-slam
- Repository: https://github.com/mit-spark/vggt-slam

## Overview

#### Purpose and entry points
VGGT-SLAM 2.0 performs real-time dense RGB scene reconstruction/SLAM from image sequences or a RealSense stream. Offline execution begins at `main.py`; live camera operation begins at `main_realtime.py`. Both orchestrate the `vggt_slam` package and can publish incremental maps to Viser.

#### Input and ingestion boundary
Offline mode consumes an ordered image folder; realtime mode consumes RealSense frames. Camera representation and pose/intrinsics handling are centralized in `vggt_slam/cameras.py`. The ingestion boundary feeds RGB observations into the mapping pipeline rather than exposing a web or service API.

#### Feed-forward reconstruction and local submaps
Frames are grouped into bounded local reconstruction units, implemented by `vggt_slam/submap.py`. A submap is the principal unit for dense geometry, camera estimates, map updates, and configurable update frequency; reducing submap size gives more frequent live updates. Image-pair/frame overlap logic in `vggt_slam/frame_overlap.py` determines useful associations within or across these local units.

#### Global map and graph state
The persistent reconstruction is managed in `vggt_slam/map.py`, while inter-submap/camera relations are represented by `vggt_slam/graph.py`. This separates dense local outputs from globally optimized state: new submaps add observations and constraints, then corrected global poses propagate to map placement.

#### Loop closure and optimization
Loop candidates and closure constraints are handled in `vggt_slam/loop_closure.py`. Global geometric optimization is implemented in `vggt_slam/solver.py`, with scale-specific estimation in `vggt_slam/scale_solver.py`. The control flow is: build local submaps → detect limited loop closures → add graph constraints → optimize global poses/scales → update the assembled map.

#### Visualization and optional semantic detection
`vggt_slam/viewer.py` publishes cameras, frustums, images, point clouds, and optional 3D detections to Viser. Visualization is downstream of mapping and can voxel-sparsify only transmitted display points. Optional open-set 3D object detection is enabled from the entry points and relies on third-party Perception Encoder and SAM 3 dependencies installed by `setup.sh`; it is not required for SLAM.

#### Runtime and external dependencies
The runtime is Python 3.11 with dependencies declared in `requirements.txt` and installation/bootstrap logic in `setup.sh`. Core reconstruction depends on the project’s VGGT fork; optimization incorporates SL(4)-oriented graph solving and likely GTSAM support. External operational dependencies include RealSense hardware for `main_realtime.py`, Viser for visualization, and ffmpeg only for converting video captures into offline image sequences.

Main entry points: [`vggt_slam/submap.py`](https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/submap.py) (Local submaps), [`vggt_slam/graph.py`](https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/graph.py) (Pose graph), [`vggt_slam/solver.py`](https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/solver.py) (Global solver), [`main.py`](https://github.com/mit-spark/vggt-slam/blob/main/main.py) (Offline runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry &amp; Inputs"]
  node_offline["Offline runner<br/>Python entry point<br/>[main.py]"]
  node_realtime["Realtime runner<br/>Python entry point<br/>[main_realtime.py]"]
  node_images["Image sequence<br/>offline input"]
  node_realsense{{"RealSense camera<br/>hardware input"}}
end

subgraph group_slam["SLAM Core"]
  node_cameras["Camera handling<br/>camera model<br/>[cameras.py]"]
  node_submaps["Local submaps<br/>dense reconstruction unit<br/>[submap.py]"]
  node_overlap["Frame overlap<br/>association logic<br/>[frame_overlap.py]"]
  node_vggt{{"VGGT fork<br/>reconstruction dependency"}}
end

subgraph group_global["Global Consistency"]
  node_map[("Persistent map<br/>dense global map<br/>[map.py]")]
  node_graph[("Pose graph<br/>constraint state<br/>[graph.py]")]
  node_loops["Loop closure<br/>constraint generation<br/>[loop_closure.py]"]
  node_solver["Global solver<br/>pose optimization<br/>[solver.py]"]
  node_scale["Scale solver<br/>scale estimation<br/>[scale_solver.py]"]
end

subgraph group_output["Outputs &amp; Optional Features"]
  node_viewer["Viser publisher<br/>visualization output<br/>[viewer.py]"]
  node_detection["Open-set 3D detection<br/>optional semantic stage<br/>[setup.sh]"]
  node_runtime["Python runtime<br/>runtime dependencies<br/>[requirements.txt]"]
end

node_images -->|"input"| node_offline
node_realsense -->|"frames"| node_realtime
node_offline -->|"orchestrates"| node_cameras
node_realtime -->|"orchestrates"| node_cameras
node_cameras -->|"RGB observations"| node_submaps
node_overlap -->|"associations"| node_submaps
node_submaps -->|"reconstructs with"| node_vggt
node_vggt -->|"dense local outputs"| node_submaps
node_submaps -->|"observations and geometry"| node_map
node_submaps -->|"poses and constraints"| node_graph
node_overlap -->|"cross-unit matches"| node_loops
node_graph -->|"search context"| node_loops
node_loops -->|"closure constraints"| node_graph
node_graph -->|"graph state"| node_solver
node_scale -->|"scale estimates"| node_solver
node_solver -->|"corrected poses"| node_graph
node_solver -->|"corrected placement"| node_map
node_map -->|"incremental map"| node_viewer
node_detection -.->|"optional 3D detections"| node_viewer
node_runtime -.->|"runs"| node_offline
node_runtime -.->|"runs"| node_realtime

click node_offline "https://github.com/mit-spark/vggt-slam/blob/main/main.py"
click node_realtime "https://github.com/mit-spark/vggt-slam/blob/main/main_realtime.py"
click node_cameras "https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/cameras.py"
click node_submaps "https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/submap.py"
click node_overlap "https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/frame_overlap.py"
click node_map "https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/map.py"
click node_graph "https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/graph.py"
click node_loops "https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/loop_closure.py"
click node_solver "https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/solver.py"
click node_scale "https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/scale_solver.py"
click node_viewer "https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/viewer.py"
click node_detection "https://github.com/mit-spark/vggt-slam/blob/main/setup.sh"
click node_runtime "https://github.com/mit-spark/vggt-slam/blob/main/requirements.txt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_offline,node_realtime,node_images,node_realsense toneBlue
class node_cameras,node_submaps,node_overlap,node_vggt toneAmber
class node_map,node_graph,node_loops,node_solver,node_scale toneMint
class node_viewer,node_detection,node_runtime toneRose
```

## Components

### Entry & Inputs

- [Offline runner](https://github.com/mit-spark/vggt-slam/blob/main/main.py): Python entry point · Processes ordered image folders. · `main.py`
- [Realtime runner](https://github.com/mit-spark/vggt-slam/blob/main/main_realtime.py): Python entry point · Processes live RealSense frames. · `main_realtime.py`
- **Image sequence**: offline input
- **RealSense camera**: hardware input

### SLAM Core

- [Camera handling](https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/cameras.py): camera model · Centralizes poses and intrinsics. · `vggt_slam/cameras.py`
- [Local submaps](https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/submap.py): dense reconstruction unit · Bounded units for geometry, cameras, and map updates. · `vggt_slam/submap.py`
- [Frame overlap](https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/frame_overlap.py): association logic · Selects useful within- and cross-submap associations. · `vggt_slam/frame_overlap.py`
- **VGGT fork**: reconstruction dependency · Provides core feed-forward reconstruction.

### Global Consistency

- [Persistent map](https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/map.py): dense global map · Assembles local reconstruction outputs. · `vggt_slam/map.py`
- [Pose graph](https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/graph.py): constraint state · Stores camera and inter-submap relations. · `vggt_slam/graph.py`
- [Loop closure](https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/loop_closure.py): constraint generation · Finds limited loop candidates and closures. · `vggt_slam/loop_closure.py`
- [Global solver](https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/solver.py): pose optimization · Optimizes graph geometry and global poses. · `vggt_slam/solver.py`
- [Scale solver](https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/scale_solver.py): scale estimation · `vggt_slam/scale_solver.py`

### Outputs & Optional Features

- [Viser publisher](https://github.com/mit-spark/vggt-slam/blob/main/vggt_slam/viewer.py): visualization output · Publishes incremental cameras, imagery, and point clouds. · `vggt_slam/viewer.py`
- [Open-set 3D detection](https://github.com/mit-spark/vggt-slam/blob/main/setup.sh): optional semantic stage · Uses Perception Encoder and SAM 3. · `setup.sh`
- [Python runtime](https://github.com/mit-spark/vggt-slam/blob/main/requirements.txt): runtime dependencies · Python 3.11 dependencies and bootstrap installation. · `requirements.txt`

## Connections

- Image sequence → Offline runner: input
- RealSense camera → Realtime runner: frames
- Offline runner → Camera handling: orchestrates
- Realtime runner → Camera handling: orchestrates
- Camera handling → Local submaps: RGB observations
- Frame overlap → Local submaps: associations
- Local submaps → VGGT fork: reconstructs with
- VGGT fork → Local submaps: dense local outputs
- Local submaps → Persistent map: observations and geometry
- Local submaps → Pose graph: poses and constraints
- Frame overlap → Loop closure: cross-unit matches
- Pose graph → Loop closure: search context
- Loop closure → Pose graph: closure constraints
- Pose graph → Global solver: graph state
- Scale solver → Global solver: scale estimates
- Global solver → Pose graph: corrected poses
- Global solver → Persistent map: corrected placement
- Persistent map → Viser publisher: incremental map
- Open-set 3D detection → Viser publisher: optional 3D detections
- Python runtime → Offline runner: runs
- Python runtime → Realtime runner: runs
