How willyzw/hi-slam2 works
Project type and purpose
• HI-SLAM2 is a research-grade, full-pipeline monocular SLAM system that reconstructs dense 3D geometry using a Gaussian-splatting map representation.
• It integrates deep-learning priors (MiDaS/Omnidata), optical-flow-based tracking (DROID), non-linear optimization, and GPU-accelerated Gaussian splatting rendering.
File structure
• Top-level entry points and scripts:
• demo.py, tsdf_integrate.py
• scripts/ (data download, preprocessing, evaluation, run scripts)
• setup.py + environment.yaml (build and dependencies)
• Core Python package: hislam2/
• modules/: neural nets, correlation, feature extraction, GRUs
• midas/: depth/normal prediction models
• gaussian/: Gaussian-map data structures, renderer, OpenGL GUI
• geom/: pose/BA utilities, projection ops
• track_frontend.py, track_backend.py, factor_graph.py, motion_filter.py
• gs_backend.py (Gaussian-map backend)
• Native extensions: src/*.cu, *.cpp (CUDA kernels for correlations, DROID, custom splatting)
• Third-party C++ libs: diff-gaussian-rasterization, eigen, lietorch, simple-knn
Extract architecture insights from the README
• Four major phases: Data Preparation → Real-time SLAM Reconstruction → Post-processing (TSDF integration) → Evaluation
• Runtime dependencies: CUDA-11.8 PyTorch, custom CUDA kernels, OpenGL for SLAM GUI
Diagramming the system
1. Data Input & Preprocessing
• scripts/preprocess_*.py (COLMAP, frame extraction)
• calib/ & config/
2. Depth & Normal Priors
• hislam2/midas (Omnidata MiDaS models)
3. Frontend Tracking
• hislam2/track_frontend.py
• DROID network (hislam2/modules + src correlation kernels)
4. Backend Optimization & Mapping
• hislam2/track_backend.py, factor_graph.py, motion_filter.py
• hislam2/gs_backend.py (Gaussian map builder)
5. Gaussian Splatting Renderer & GUI
• hislam2/gaussian/gui/slam_gui.py
• OpenGL shaders (gau_frag.glsl, gau_vert.glsl)
6. Post-processing
• tsdf_integrate.py (mesh extraction)
7. Evaluation
• scripts/run_replica.py, run_scannet.py, eval_recon
• Raw images → Preprocessing → undistorted, intrinsics →
• Depth/normal network → Priors →
• Feature extractor & correlation kernels → Frame-to-frame flow → Frontend produces keyframes/initial poses →
• Backend consumes keyframes + priors → Factor graph optimize poses & updates Gaussian map →
• Gaussian map → Renderer (SLAM GUI) and optional TSDF integrator → Outputs: camera trajectories, dense meshes, renderings
• Evaluation scripts consume outputs + ground-truth
c. Architectural patterns & design principles
• Modular separation: data prep, neural-net priors, tracking, mapping, rendering
• GPU acceleration: heavy lifting (depth net, corr kernels, splatting) offloaded to CUDA
• Factor-graph optimization in backend for global consistency
• Clear extensibility: config/ and pretrained_models/ for swapping datasets and models
d. Key technologies & libraries
• Python: main control flow, glue code
• PyTorch: MiDaS depth/normal, DROID-Net
• CUDA/C++: correlation kernels (src/*.cu), custom rasterization (thirdparty)
• OpenGL (via glfw/GL) for real-time GUI rendering