# jonathanzhc/scenepredictor architecture

> Architecture diagram of the GitHub repository jonathanzhc/scenepredictor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jonathanzhc/scenepredictor
- Repository: https://github.com/jonathanzhc/scenepredictor

## Overview

ScenePredictor turns synchronized multi-camera RGB-D frames into tracked scene-flow and dense per-point velocities, then publishes results to ROS 2 for visualization or downstream use. Its central workflow runs the in-process tracker, selects instances present in adjacent frames, performs one DifFlow3D inference, and recovers velocity conditioned on track identity. The README documents tracker internals and optional Isaac Sim input; sampled parent-repository excerpts chiefly confirm orchestration, ROS I/O, and recovery boundaries.

Main entry points: [`scene_pred_pipeline/pipeline.py`](https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/pipeline.py) (Prediction pipeline), [`MultiViewRGBDTracker`](https://github.com/jonathanzhc/scenepredictor/tree/main/MultiViewRGBDTracker) (Multi-view tracker), [`scripts/run_scene_pred_pipeline.py`](https://github.com/jonathanzhc/scenepredictor/blob/main/scripts/run_scene_pred_pipeline.py) (ROS runtime node), [`scene_pred_pipeline/ros_input.py`](https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/ros_input.py) (ROS frame sync)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Input and orchestration"]
  node_ros_input["ROS frame sync<br/>[ros_input.py]"]
  node_runtime_node["ROS runtime node"]
  node_pipeline["Prediction pipeline<br/>[pipeline.py]"]
  node_config["Runtime configuration<br/>[config.py]"]
end

subgraph group_tracking["Multi-view tracking"]
  node_tracker_adapter["Tracker adapter<br/>[tracker_adapter.py]"]
  node_tracker_engine["Multi-view tracker"]
  node_sam3["SAM3 refresh"]
  node_efficienttam["EfficientTAM propagation"]
  node_tracker_geometry["GPU mask and geometry"]
  node_view_fusion["Cross-view fusion"]
  node_temporal_ids[("Persistent global IDs")]
end

subgraph group_motion["Scene motion"]
  node_instance_filter["Common-instance pairing<br/>[instance_filter.py]"]
  node_difflow["DifFlow3D inference<br/>[flow_prediction.py]"]
  node_difflow_runtime["DifFlow3D runtime"]
  node_velocity_recovery["Dense velocity recovery"]
end

subgraph group_publish["Output and diagnostics"]
  node_profiler["Pipeline profiling<br/>[profiler.py]"]
  node_publisher_worker["Async publisher"]
  node_ros_output["ROS visualization output<br/>[ros_output.py]"]
end

node_rgbd_sources(("RGB-D sources"))
node_isaac_sim(("Isaac Sim"))
node_ros_consumers(("ROS 2 and RViz"))

node_rgbd_sources -->|"publish frames"| node_ros_input
node_isaac_sim -.->|"publish frames"| node_ros_input
node_runtime_node -->|"loads settings"| node_config
node_runtime_node -->|"constructs input"| node_ros_input
node_ros_input -->|"delivers frames"| node_runtime_node
node_runtime_node -->|"processes frames"| node_pipeline
node_pipeline -->|"processes frame"| node_tracker_adapter
node_tracker_adapter -->|"invokes tracker"| node_tracker_engine
node_tracker_engine -.->|"refreshes prompts"| node_sam3
node_tracker_engine -->|"propagates masks"| node_efficienttam
node_tracker_engine -->|"processes masks"| node_tracker_geometry
node_tracker_engine -->|"fuses views"| node_view_fusion
node_tracker_engine -->|"maintains IDs"| node_temporal_ids
node_pipeline -->|"selects shared tracks"| node_instance_filter
node_pipeline -->|"predicts motion"| node_difflow
node_difflow -->|"runs inference"| node_difflow_runtime
node_pipeline -->|"recovers velocity"| node_velocity_recovery
node_difflow -->|"provides anchors"| node_velocity_recovery
node_pipeline -->|"records timings"| node_profiler
node_runtime_node -->|"queues output"| node_publisher_worker
node_publisher_worker -->|"publishes results"| node_ros_output
node_ros_output -->|"publishes topics"| node_ros_consumers

click node_ros_input "https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/ros_input.py"
click node_runtime_node "https://github.com/jonathanzhc/scenepredictor/blob/main/scripts/run_scene_pred_pipeline.py"
click node_pipeline "https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/pipeline.py"
click node_config "https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/config.py"
click node_tracker_adapter "https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/tracker_adapter.py"
click node_tracker_engine "https://github.com/jonathanzhc/scenepredictor/tree/main/MultiViewRGBDTracker"
click node_sam3 "https://github.com/jonathanzhc/scenepredictor/tree/main/MultiViewRGBDTracker"
click node_efficienttam "https://github.com/jonathanzhc/scenepredictor/tree/main/MultiViewRGBDTracker"
click node_tracker_geometry "https://github.com/jonathanzhc/scenepredictor/tree/main/MultiViewRGBDTracker"
click node_view_fusion "https://github.com/jonathanzhc/scenepredictor/tree/main/MultiViewRGBDTracker"
click node_temporal_ids "https://github.com/jonathanzhc/scenepredictor/tree/main/MultiViewRGBDTracker"
click node_instance_filter "https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/instance_filter.py"
click node_difflow "https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/flow_prediction.py"
click node_difflow_runtime "https://github.com/jonathanzhc/scenepredictor/tree/main/DifFlow3D"
click node_velocity_recovery "https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/velocity_recovery.py"
click node_profiler "https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/profiler.py"
click node_publisher_worker "https://github.com/jonathanzhc/scenepredictor/blob/main/scripts/run_scene_pred_pipeline.py"
click node_ros_output "https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/ros_output.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ros_input,node_runtime_node,node_pipeline,node_config toneBlue
class node_tracker_adapter,node_tracker_engine,node_sam3,node_efficienttam,node_tracker_geometry,node_view_fusion,node_temporal_ids toneAmber
class node_instance_filter,node_difflow,node_difflow_runtime,node_velocity_recovery toneMint
class node_profiler,node_publisher_worker,node_ros_output toneRose
class node_rgbd_sources,node_isaac_sim,node_ros_consumers toneIndigo
```

## Components

### Input and orchestration

- [ROS frame sync](https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/ros_input.py): `scene_pred_pipeline/ros_input.py`
- [ROS runtime node](https://github.com/jonathanzhc/scenepredictor/blob/main/scripts/run_scene_pred_pipeline.py): `scripts/run_scene_pred_pipeline.py`
- [Prediction pipeline](https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/pipeline.py): `scene_pred_pipeline/pipeline.py`
- [Runtime configuration](https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/config.py): `scene_pred_pipeline/config.py`

### Multi-view tracking

- [Tracker adapter](https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/tracker_adapter.py): `scene_pred_pipeline/tracker_adapter.py`
- [Multi-view tracker](https://github.com/jonathanzhc/scenepredictor/tree/main/MultiViewRGBDTracker): `MultiViewRGBDTracker`
- [SAM3 refresh](https://github.com/jonathanzhc/scenepredictor/tree/main/MultiViewRGBDTracker): `MultiViewRGBDTracker`
- [EfficientTAM propagation](https://github.com/jonathanzhc/scenepredictor/tree/main/MultiViewRGBDTracker): `MultiViewRGBDTracker`
- [GPU mask and geometry](https://github.com/jonathanzhc/scenepredictor/tree/main/MultiViewRGBDTracker): `MultiViewRGBDTracker`
- [Cross-view fusion](https://github.com/jonathanzhc/scenepredictor/tree/main/MultiViewRGBDTracker): `MultiViewRGBDTracker`
- [Persistent global IDs](https://github.com/jonathanzhc/scenepredictor/tree/main/MultiViewRGBDTracker): `MultiViewRGBDTracker`

### Scene motion

- [Common-instance pairing](https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/instance_filter.py): `scene_pred_pipeline/instance_filter.py`
- [DifFlow3D inference](https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/flow_prediction.py): `scene_pred_pipeline/flow_prediction.py`
- [DifFlow3D runtime](https://github.com/jonathanzhc/scenepredictor/tree/main/DifFlow3D): `DifFlow3D`
- [Dense velocity recovery](https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/velocity_recovery.py): `scene_pred_pipeline/velocity_recovery.py`

### Output and diagnostics

- [Pipeline profiling](https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/profiler.py): `scene_pred_pipeline/profiler.py`
- [Async publisher](https://github.com/jonathanzhc/scenepredictor/blob/main/scripts/run_scene_pred_pipeline.py): `scripts/run_scene_pred_pipeline.py`
- [ROS visualization output](https://github.com/jonathanzhc/scenepredictor/blob/main/scene_pred_pipeline/ros_output.py): `scene_pred_pipeline/ros_output.py`

### Other components

- **RGB-D sources**
- **Isaac Sim**
- **ROS 2 and RViz**

## Connections

- RGB-D sources → ROS frame sync: publish frames
- Isaac Sim → ROS frame sync: publish frames
- ROS runtime node → Runtime configuration: loads settings
- ROS runtime node → ROS frame sync: constructs input
- ROS frame sync → ROS runtime node: delivers frames
- ROS runtime node → Prediction pipeline: processes frames
- Prediction pipeline → Tracker adapter: processes frame
- Tracker adapter → Multi-view tracker: invokes tracker
- Multi-view tracker → SAM3 refresh: refreshes prompts
- Multi-view tracker → EfficientTAM propagation: propagates masks
- Multi-view tracker → GPU mask and geometry: processes masks
- Multi-view tracker → Cross-view fusion: fuses views
- Multi-view tracker → Persistent global IDs: maintains IDs
- Prediction pipeline → Common-instance pairing: selects shared tracks
- Prediction pipeline → DifFlow3D inference: predicts motion
- DifFlow3D inference → DifFlow3D runtime: runs inference
- Prediction pipeline → Dense velocity recovery: recovers velocity
- DifFlow3D inference → Dense velocity recovery: provides anchors
- Prediction pipeline → Pipeline profiling: records timings
- ROS runtime node → Async publisher: queues output
- Async publisher → ROS visualization output: publishes results
- ROS visualization output → ROS 2 and RViz: publishes topics