How jonathanzhc/scenepredictor works
ScenePredictor turns synchronized multi-camera RGB-D frames into tracked scene-flow and dense per-point velocities, then publishes results to ROS 2 for visualization or downstream use. Its central workflow runs the in-process tracker, selects instances present in adjacent frames, performs one DifFlow3D inference, and recovers velocity conditioned on track identity. The README documents tracker internals and optional Isaac Sim input; sampled parent-repository excerpts chiefly confirm orchestration, ROS I/O, and recovery boundaries.
Main entry points: pipeline.py Prediction pipeline · MultiViewRGBDTracker Multi-view tracker · run_scene_pred_pipeline.py ROS runtime node · ros_input.py ROS frame sync
Components 21
Input and orchestration
- ROS frame sync
scene_pred_pipeline/ros_input.py - ROS runtime node
scripts/run_scene_pred_pipeline.py - Prediction pipeline
scene_pred_pipeline/pipeline.py - Runtime configuration
scene_pred_pipeline/config.py
Multi-view tracking
- Tracker adapter
scene_pred_pipeline/tracker_adapter.py - Multi-view tracker
MultiViewRGBDTracker - SAM3 refresh
MultiViewRGBDTracker - EfficientTAM propagation
MultiViewRGBDTracker - GPU mask and geometry
MultiViewRGBDTracker - Cross-view fusion
MultiViewRGBDTracker - Persistent global IDs
MultiViewRGBDTracker
Scene motion
- Common-instance pairing
scene_pred_pipeline/instance_filter.py - DifFlow3D inference
scene_pred_pipeline/flow_prediction.py - DifFlow3D runtime
DifFlow3D - Dense velocity recovery
scene_pred_pipeline/velocity_recovery.py
Output and diagnostics
- Pipeline profiling
scene_pred_pipeline/profiler.py - Async publisher
scripts/run_scene_pred_pipeline.py - ROS visualization output
scene_pred_pipeline/ros_output.py
Other components
- RGB-D sources
- Isaac Sim
- ROS 2 and RViz
Connections 22
- RGB-D sources → ROS frame sync: publish frames
- Isaac Sim → ROS frame sync: publish frames
- ROS runtime node → Runtime configuration: loads settings
- ROS runtime node → ROS frame sync: constructs input
- ROS frame sync → ROS runtime node: delivers frames
- ROS runtime node → Prediction pipeline: processes frames
- Prediction pipeline → Tracker adapter: processes frame
- Tracker adapter → Multi-view tracker: invokes tracker
- Multi-view tracker → SAM3 refresh: refreshes prompts
- Multi-view tracker → EfficientTAM propagation: propagates masks
- Multi-view tracker → GPU mask and geometry: processes masks
- Multi-view tracker → Cross-view fusion: fuses views
- Multi-view tracker → Persistent global IDs: maintains IDs
- Prediction pipeline → Common-instance pairing: selects shared tracks
- Prediction pipeline → DifFlow3D inference: predicts motion
- DifFlow3D inference → DifFlow3D runtime: runs inference
- Prediction pipeline → Dense velocity recovery: recovers velocity
- DifFlow3D inference → Dense velocity recovery: provides anchors
- Prediction pipeline → Pipeline profiling: records timings
- ROS runtime node → Async publisher: queues output
- Async publisher → ROS visualization output: publishes results
- ROS visualization output → ROS 2 and RViz: publishes topics