raidhu/object_projection_3d

Purpose and entry points

This is a ROS 2 package that turns upstream 2D detectNet detections plus aligned RealSense depth into oriented 3D boxes, normally expressed in map. It does not own the camera or 2D inference. The runtime entry point is the Localizer3D ROS node in rgbd3d/ros2_node.py, packaged through setup.py and package.xml. viz_node.py is a separate visualization-oriented entry point.

ROS boundary and frame lifecycle

Localizer3D is the sole ROS-facing layer: it subscribes to Detection2DArray, aligned compressed/raw depth, camera intrinsics, and TF; it publishes Detection3DArray, RViz markers, and optional frustum clouds. It buffers depth frames and pairs each detection message with the nearest acceptable depth timestamp rather than using strict synchronization. TF transforms decoded poses from the camera optical frame to target_frame (map by default); failed lookups deliberately retain camera-frame output.

Paths: rgbd3d/ros2_node.py, rgbd3d/box_to_ros.py, rgbd3d/config.py.

Detection admission and duplicate control

Incoming 2D hypotheses are normalized to class names/IDs, filtered by confidence and the supported car/person allow-list, then reduced with greedy 2D suppression. Suppression considers conventional IoU and containment coverage, which handles nested detectNet boxes. After 3D inference, a ground-plane proximity check suppresses duplicate boxes that represent the same object.

Paths: rgbd3d/ros2_node.py, rgbd3d/dedupe.py, rgbd3d/config.py.

Depth-to-frustum geometry

For every retained 2D ROI, the pipeline validates depth values, clamps image bounds, deprojects valid pixels using calibrated pinhole intrinsics, and rejects sparse clouds. It estimates an object-specific scale from apparent image height and median depth, then rotates the frustum toward the network’s canonical forward view. This is the key boundary between camera/depth geometry and model-ready point clouds.

Paths: rgbd3d/frustum.py, rgbd3d/ros2_node.py, rgbd3d/fp_constants.py.

Two-stage Frustum-PointNet inference

FrustumPointNet3D runs a split Frustum-PointNet v1 pipeline: stage A produces point segmentation logits; host-side NumPy masking selects and samples foreground points; stage B regresses center, heading bins/residuals, and size clusters/residuals. TensorRT is the intended accelerated backend, with lazy engine setup/buffer management; ONNX Runtime provides a portable offline fallback. The split is architectural because the original masking operation cannot be exported to ONNX.

Paths: rgbd3d/model3d.py, rgbd3d/box_decode.py, tools/export_frustum_pointnet.py.

Metric-box reconstruction and outputs

Decoded model outputs are re-centered on the segmented foreground centroid, unrotated from canonical frustum coordinates, and divided by the per-object scale to recover camera-frame metres. Size and orientation come from KITTI-trained model priors rather than geometric extents/PCA; class-to-size-cluster selection is therefore a material accuracy constraint for model-scale vehicles. The ROS layer converts this result into pose/size messages, transforms pose only across frames, and emits markers; dense raw frustum clouds are generated only if subscribed.

Paths: rgbd3d/box_decode.py, rgbd3d/box_to_ros.py, rgbd3d/visualize.py.

Runtime dependencies and operational constraints

The deployment graph requires a RealSense source with depth aligned to color, an external detectNet ROS graph, and TF from camera to robot/localization map. TensorRT/CUDA is the performance path; ONNX Runtime supports offline validation. The node uses BEST_EFFORT depth/detection QoS by default to avoid Wi-Fi retransmission backlog and pins BLAS/OpenMP threading to avoid starving the single-threaded ROS executor.

Paths: rgbd3d/ros2_node.py, rgbd3d/model3d.py, package.xml.