# arghyachatterjee/mast3r-slam architecture

> Architecture diagram of the GitHub repository arghyachatterjee/mast3r-slam, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arghyachatterjee/mast3r-slam
- Repository: https://github.com/arghyachatterjee/mast3r-slam

## Overview

##### Project Type and Purpose
- This is a research-grade full-stack SLAM (Simultaneous Localization and Mapping) system named MASt3R-SLAM, combining classical geometry with deep 3D‐reconstruction priors.
- It ingests RGB/D video or live camera streams, runs a real-time dense SLAM pipeline (frame preprocessing → tracking → optimization), integrates pretrained MASt3R models for learned priors, and visualizes results while supporting batch evaluation against ground-truth datasets.

##### High-Level Components
a. Data Ingestion
- Input Sources:
– Datasets on disk (TUM, 7-Scenes, EuRoC, ETH3D).
– MP4 videos or image folders.
– Live RealSense camera stream.
- Scripts: download_*.sh for retrieving datasets; main.py CLI wrapper.
b. Configuration
- YAML configs (config/*.yaml) for dataset paths, intrinsics, calibration modes.
- config.py summarizes and exposes parameters.
c. Core SLAM Pipeline (Python modules under mast3r_slam/)
- dataloader.py: frames ↔ timestamped RGB/D.
- image.py, frame.py: low-level image and frame abstractions.
- matching.py, retrieval_database.py: candidate frame retrieval & feature matching.
- tracker.py: local pose tracking loop.
- global_opt.py & nonlinear_optimizer.py: pose-graph and geometry optimization.
- geometry.py, mast3r_utils.py, lietorch_utils.py: SE(3) math, Lie algebra helpers.
- evaluate.py / scripts/eval_*.sh: headless batch evaluation against ground truth.
d. Backend Compute Kernels (C++/CUDA library)
- mast3r_slam/backend/: GN solver & matching routines in gn.cpp / gn_kernels.cu / matching_kernels.cu exposed via PyBind11.
e. Deep Reconstruction Prior
- thirdparty/mast3r/: pretrained MASt3R network for dense reconstruction and learned feature extraction.
- in3d: GPU‐accelerated 3D visualization & OpenGL window management.
f. Visualization
- visualization.py & visualization_utils.py: real‐time 2D/3D map display, overlays using ModernGL via in3d.
- media/: sample gifs/images for README.
g. Checkpoints & Ground Truth
- checkpoints/: pretrained model weights (downloaded).
- groundtruths/: stored poses for evaluation.

##### Relationships and Data Flow
1. User runs main.py → CLI parses config and dataset path → Initialize dataloader.
2. Dataloader feeds frames to tracker; frame.py/image.py perform basic processing.
3. For each new frame:
- Retrieval DB queries past keyframes (retrieval_database.py).
- matching.py dispatches matching_kernels (CUDA) for feature correspondences.
- Frame-to-model pose update via GN solver (gn_kernels + gn.cpp).
- Optionally call MASt3R deep model (thirdparty/mast3r) for learned depth/prior.
4. Tracker accumulates poses → global optimizer triggers periodic pose‐graph optimization via nonlinear_optimizer.py & global_opt.py.
5. Visualization module (in3d based) renders camera trajectory, reconstructed surfaces in real time.
6. Upon completion or evaluation mode, evaluate.py compares estimates with groundtruths and logs metrics.

##### Architectural Patterns & Design Principles
- Layered Architecture:
- CLI / orchestration (main.py)
- Configuration & dataset I/O
- Algorithmic layer (Python modules)
- Performance layer (C++/CUDA kernels)
- Third-party deep models & rendering libraries
- Separation of Concerns: matching, tracking, optimization, visualization, evaluation.
- Extensibility via submodules (mast3r/in3d) and config-driven behavior.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Ingestion Layer
    subgraph "Data Ingestion" 
        direction TB
        Main["main.py"]:::python
        Downloads["Download Scripts"]:::python
        Datasets["Datasets & Live Stream"]:::storage
        Main -->|"invokes"| Downloads
        Main -->|"reads from"| Datasets
    end

    %% Configuration Layer
    subgraph "Configuration" 
        direction TB
        ConfigYAML["YAML Configs"]:::storage
        ConfigLoader["mast3r_slam/config.py"]:::python
        ConfigLoader -->|"parses"| ConfigYAML
    end

    %% Core SLAM Pipeline
    subgraph "Core SLAM Pipeline" 
        direction TB
        Dataloader["mast3r_slam/dataloader.py"]:::python
        FrameImg["mast3r_slam/frame.py & mast3r_slam/image.py"]:::python
        Matching["mast3r_slam/matching.py & retrieval_database.py"]:::python
        Tracker["mast3r_slam/tracker.py"]:::python
        GlobalOpt["mast3r_slam/global_opt.py"]:::python
        Nonlinear["mast3r_slam/nonlinear_optimizer.py"]:::python
        Evaluate["mast3r_slam/evaluate.py"]:::python
        Utils["mast3r_slam/geometry.py, mast3r_utils.py, lietorch_utils.py"]:::python
        Datasets -->|“frames”| Dataloader
        Dataloader -->|“FrameData”| FrameImg
        FrameImg -->|“images”| Matching
        Matching -->|“matches”| Tracker
        Tracker -->|“poses”| GlobalOpt
        GlobalOpt -->|“optimized poses”| Tracker
        Tracker -->|“to eval”| Evaluate
        Tracker -->|“uses”| Nonlinear
        Tracker -->|“uses”| Utils
    end

    %% Backend Compute Kernels
    subgraph "Backend (C++/CUDA)" 
        direction TB
        GNHeader["gn.h"]:::cpp
        GNSolver["gn.cpp & gn_kernels.cu"]:::cpp
        MatchK["matching_kernels.cu"]:::cpp
        Tracker -->|"calls via PyBind11"| GNSolver
        Matching -->|"CUDA kernels"| MatchK
    end

    %% Deep Reconstruction Priors
    subgraph "Deep Models & Rendering" 
        direction TB
        MASt3R["thirdparty/mast3r/"]:::dl
        In3D["thirdparty/in3d/"]:::dl
        Tracker -->|"optional prior"| MASt3R
        Visualization -->|"uses"| In3D
    end

    %% Visualization Layer
    subgraph "Visualization" 
        direction TB
        VizPy["mast3r_slam/visualization.py & visualization_utils.py"]:::vis
        Shaders["resources/programs/*.glsl"]:::vis
        Media["media/"]:::vis
        Tracker -->|"stream data"| VizPy
        VizPy -->|"loads shaders"| Shaders
        VizPy -->|"sample media"| Media
    end

    %% Storage & Evaluation Data
    subgraph "Storage" 
        direction TB
        GroundTruth["groundtruths/"]:::storage
        Checkpoints["checkpoints/"]:::storage
        Evaluate -->|"compares with"| GroundTruth
        MASt3R -->|"loads weights from"| Checkpoints
    end

    %% Hardware
    GPU["GPU (CUDA)"]:::gpu
    GNSolver -->|"executes on"| GPU
    MatchK -->|"executes on"| GPU
    MASt3R -->|"executes on"| GPU

    %% Click Events
    click Main "https://github.com/arghyachatterjee/mast3r-slam/blob/main/main.py"
    click Downloads "https://github.com/arghyachatterjee/mast3r-slam/blob/main/scripts/download_tum.sh"
    click Downloads "https://github.com/arghyachatterjee/mast3r-slam/blob/main/scripts/download_7_scenes.sh"
    click Downloads "https://github.com/arghyachatterjee/mast3r-slam/blob/main/scripts/download_euroc.sh"
    click Downloads "https://github.com/arghyachatterjee/mast3r-slam/blob/main/scripts/download_eth3d.sh"
    click Datasets "https://github.com/arghyachatterjee/mast3r-slam/tree/main/groundtruths/"
    click ConfigYAML "https://github.com/arghyachatterjee/mast3r-slam/blob/main/config/base.yaml"
    click ConfigYAML "https://github.com/arghyachatterjee/mast3r-slam/blob/main/config/calib.yaml"
    click ConfigYAML "https://github.com/arghyachatterjee/mast3r-slam/blob/main/config/eth3d.yaml"
    click ConfigYAML "https://github.com/arghyachatterjee/mast3r-slam/blob/main/config/eval_calib.yaml"
    click ConfigYAML "https://github.com/arghyachatterjee/mast3r-slam/blob/main/config/eval_no_calib.yaml"
    click ConfigYAML "https://github.com/arghyachatterjee/mast3r-slam/blob/main/config/intrinsics.yaml"
    click ConfigLoader "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/config.py"
    click Dataloader "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/dataloader.py"
    click FrameImg "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/frame.py"
    click FrameImg "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/image.py"
    click Matching "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/matching.py"
    click Matching "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/retrieval_database.py"
    click Tracker "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/tracker.py"
    click GlobalOpt "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/global_opt.py"
    click Nonlinear "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/nonlinear_optimizer.py"
    click Evaluate "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/evaluate.py"
    click Utils "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/geometry.py"
    click Utils "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/mast3r_utils.py"
    click Utils "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/lietorch_utils.py"
    click GNHeader "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/backend/include/gn.h"
    click GNSolver "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/backend/src/gn.cpp"
    click GNSolver "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/backend/src/gn_kernels.cu"
    click MatchK "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/backend/src/matching_kernels.cu"
    click MASt3R "https://github.com/arghyachatterjee/mast3r-slam/tree/main/thirdparty/mast3r/"
    click In3D "https://github.com/arghyachatterjee/mast3r-slam/tree/main/thirdparty/in3d/"
    click VizPy "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/visualization.py"
    click VizPy "https://github.com/arghyachatterjee/mast3r-slam/blob/main/mast3r_slam/visualization_utils.py"
    click Shaders "https://github.com/arghyachatterjee/mast3r-slam/blob/main/resources/programs/surfelmap.glsl"
    click Shaders "https://github.com/arghyachatterjee/mast3r-slam/blob/main/resources/programs/trianglemap.glsl"
    click Media "https://github.com/arghyachatterjee/mast3r-slam/tree/main/media/"
    click GroundTruth "https://github.com/arghyachatterjee/mast3r-slam/tree/main/groundtruths/"
    click Checkpoints "https://github.com/arghyachatterjee/mast3r-slam/tree/main/checkpoints/"

    %% Styles
    classDef python fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef cpp fill:#ffe5b4,stroke:#333,stroke-width:1px
    classDef dl fill:#d4f4dd,stroke:#333,stroke-width:1px
    classDef storage fill:#e0e0e0,stroke:#333,stroke-width:1px
    classDef vis fill:#e5d4f4,stroke:#333,stroke-width:1px
    classDef gpu fill:#f4f4a3,stroke:#333,stroke-width:1px
```
