How princeton-vl/droid-slam works
Project Type and Purpose
• DROID-SLAM is an open-source deep visual SLAM system for monocular, stereo, and RGB-D cameras. It combines deep neural network inference, classical geometry, and optimization to produce camera trajectories and dense reconstructions.
• It provides both inference (demo.py, view_reconstruction.py) and training (train.py) pipelines, plus evaluation scripts for standard SLAM benchmarks.
File-Structure-Based Component Analysis
Top-level directories and their responsibilities:
• droid_slam/
• Core SLAM pipeline: frontend, backend, optimization, network, geometry, modules, utilities
• src/
• Custom C++/CUDA extensions (kernels for correlation, feature matching)
• thirdparty/
• External libraries: lietorch, pytorch_scatter, tartanair_tools
• data/ and calib/
• Sample datasets and calibration files
• demo.py, train.py, view_reconstruction.py
• Entry points for inference, training, and visualization
• evaluation_scripts/
• Benchmark evaluation pipelines for ETH3D, EuRoC, TUM, TartanAir
• tools/
• Shell scripts to download data and pre-trained models
• requirements*.txt, environment*.yaml, setup.py
• Environment and install specifications
Key Architectural Patterns
• Pipeline separation between data ingestion, neural network inference (DROID Net), frontend processing, backend optimization (factor graph), and visualization.
• Asynchronous multiprocessing option: frontend and backend can run in separate processes/devices.
• Modular design: pluggable data_readers, network modules, geometry utilities, and visualization backends.
• Use of custom CUDA kernels for performance-critical operations.
Main Components and Their Interactions
a. Data Ingestion Layer
• droid_slam/data_readers: Reads images, depth, inertial data, applies augmentation.
• Inputs: raw sensor data + calibration files → Outputs: batched tensors.
b. DROID Network
• droid_slam/droid_net: Computes dense optical flow, feature correpondences.
• Uses custom CUDA kernels (src/*.cu) for correlation.
• Outputs: flow fields, correlation features.
c. Frontend
• droid_slam/droid_frontend / droid_slam/droid_async:
• Ingests network outputs, extracts keypoints (module: extractor), builds initial pose estimates.
d. Backend
• droid_slam/droid_backend:
• Builds factor graph (factor_graph.py), uses graph_utils & chol for optimization.
• Applies robust losses (losses.py) and solves for camera poses.
e. SLAM Core
• Motion filtering (motion_filter.py), trajectory filling (trajectory_filler.py).
• Align (align.py) for scale/alignment to ground truth.
f. Visualization
• droid_slam/visualizer: 3D scene and camera trajectory rendering via OpenGL (moderngl).
• view_reconstruction.py: High-resolution point cloud viewer.
g. Training
• train.py: Orchestrates data loading, forward pass through DROID net, backend, loss computation, backprop.
• Caches covisibility computations.
h. Benchmark Evaluation
• evaluation_scripts/*.py and tools/*.sh: Automate dataset download, inference, compute ATE/RPE, parse results.