How shashankyld/dynamic_gs works
Overall System “Layers” and Main Components
• Frontend / Data Acquisition:
• Unified DataLoader (see “Frame”, “KeyFrame” concepts in the README).
• Input sensors: RGBD frames (and possibly other lighter modalities).
• Feature Processing and Tracking:
• Feature Detection & Matching: Using Delaunay triangulation–based matching (removing current frame Delaunay each time and tracking via the reference frame).
• Frame-to-frame, frame-to-sparse map, and image-to-GS map tracking.
• Feature Det/Match module that also collects common feature counts to help decide keyframe spawning.
• Dynamic Object Analysis:
• SAM2 integration for proposing dynamic masks.
• Adjustment/improvement of prompts based on consistency with the Delaunay feature matching.
• SLAM Backend:
• Core SLAM module that handles pose estimation, loop closure, and keyframe management.
• Factor graph optimization block (using gtsam/g20/pypose).
• TimingUtils integrated throughout to track each method’s latency.
• Mapping Subsystems:
• Sparse Map module: Maintains list of keyframes and associated metadata (features, tracked pose, dynamic detection results).
• Dense Map (GS Map) module: Manages a list of “Gaussians” (each attached to a keyframe), with operations such as spawn, prune, and densify.
• Parallel Processing and Worker Threads:
• The main thread handles feature detection, pose estimation, and core SLAM operations.
• Worker threads execute SAM2 processing, Gaussian splatting optimization, loop closure detection, and map maintenance.
• Visualization and Testing:
• Real-time trajectory visualization and map quality assessment (both interactive and batch mode).
• Testing frameworks and benchmarking modules (see tests/ and performance target benchmarks in the README).
Architectural Relationships and Data Flow
• Data Flow: Start with the DataLoader reading input frames ➔ Each frame is processed by the Feature Detector Matcher module ➔ The extracted features and matches feed into the Track and SLAM modules for pose estimation.
• Dynamic processing: Parallel to tracking, the dynamic mask generation module (powered by SAM2) receives input images and produces masks that are used both for refining prompts and gating the mapping update.
• Mapping: Once a frame qualifies as a keyframe (based on common feature counts and metric thresholds), it is pushed into the Sparse Map where its full set of data (features, pose, dynamic mask) is stored. In parallel, the Dense Map module updates its Gaussian Splatting representation using data from the keyframes.
• Factor Graph & Loop Closure: The SLAM backend sends data to a factor graph optimization module and receives corrections via loop closure detections which are handled in worker threads.
• Visualization: Both the trajectory (from pose estimation) and map quality (sparse and dense maps) are forwarded to visualization modules for real-time display and debugging.