How arghyachatterjee/mast3r-slam works
Project Type and Purpose
• This is a research-grade full-stack SLAM (Simultaneous Localization and Mapping) system named MASt3R-SLAM, combining classical geometry with deep 3D‐reconstruction priors.
• It ingests RGB/D video or live camera streams, runs a real-time dense SLAM pipeline (frame preprocessing → tracking → optimization), integrates pretrained MASt3R models for learned priors, and visualizes results while supporting batch evaluation against ground-truth datasets.
High-Level Components
a. Data Ingestion
• Input Sources:
– Datasets on disk (TUM, 7-Scenes, EuRoC, ETH3D).
– MP4 videos or image folders.
– Live RealSense camera stream.
• Scripts: download_*.sh for retrieving datasets; main.py CLI wrapper.
b. Configuration
• YAML configs (config/*.yaml) for dataset paths, intrinsics, calibration modes.
• config.py summarizes and exposes parameters.
c. Core SLAM Pipeline (Python modules under mast3r_slam/)
• dataloader.py: frames ↔ timestamped RGB/D.
• image.py, frame.py: low-level image and frame abstractions.
• matching.py, retrieval_database.py: candidate frame retrieval & feature matching.
• tracker.py: local pose tracking loop.
• global_opt.py & nonlinear_optimizer.py: pose-graph and geometry optimization.
• geometry.py, mast3r_utils.py, lietorch_utils.py: SE(3) math, Lie algebra helpers.
• evaluate.py / scripts/eval_*.sh: headless batch evaluation against ground truth.
d. Backend Compute Kernels (C++/CUDA library)
• mast3r_slam/backend/: GN solver & matching routines in gn.cpp / gn_kernels.cu / matching_kernels.cu exposed via PyBind11.
e. Deep Reconstruction Prior
• thirdparty/mast3r/: pretrained MASt3R network for dense reconstruction and learned feature extraction.
• in3d: GPU‐accelerated 3D visualization & OpenGL window management.
f. Visualization
• visualization.py & visualization_utils.py: real‐time 2D/3D map display, overlays using ModernGL via in3d.
• media/: sample gifs/images for README.
g. Checkpoints & Ground Truth
• checkpoints/: pretrained model weights (downloaded).
• groundtruths/: stored poses for evaluation.
Relationships and Data Flow
1. User runs main.py → CLI parses config and dataset path → Initialize dataloader.
2. Dataloader feeds frames to tracker; frame.py/image.py perform basic processing.
3. For each new frame:
• Retrieval DB queries past keyframes (retrieval_database.py).
• matching.py dispatches matching_kernels (CUDA) for feature correspondences.
• Frame-to-model pose update via GN solver (gn_kernels + gn.cpp).
• Optionally call MASt3R deep model (thirdparty/mast3r) for learned depth/prior.
4. Tracker accumulates poses → global optimizer triggers periodic pose‐graph optimization via nonlinear_optimizer.py & global_opt.py.
5. Visualization module (in3d based) renders camera trajectory, reconstructed surfaces in real time.
6. Upon completion or evaluation mode, evaluate.py compares estimates with groundtruths and logs metrics.
Architectural Patterns & Design Principles
• Layered Architecture:
• CLI / orchestration (main.py)
• Configuration & dataset I/O
• Algorithmic layer (Python modules)
• Performance layer (C++/CUDA kernels)
• Third-party deep models & rendering libraries
• Separation of Concerns: matching, tracking, optimization, visualization, evaluation.
• Extensibility via submodules (mast3r/in3d) and config-driven behavior.