# raidhu/object_projection_3d architecture

> Architecture diagram of the GitHub repository raidhu/object_projection_3d, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/raidhu/object_projection_3d
- Repository: https://github.com/raidhu/object_projection_3d

## Overview

#### Purpose and entry points
This is a ROS 2 package that turns upstream 2D detectNet detections plus aligned RealSense depth into oriented 3D boxes, normally expressed in `map`. It does not own the camera or 2D inference. The runtime entry point is the `Localizer3D` ROS node in `rgbd3d/ros2_node.py`, packaged through `setup.py` and `package.xml`. `viz_node.py` is a separate visualization-oriented entry point.

#### ROS boundary and frame lifecycle
`Localizer3D` is the sole ROS-facing layer: it subscribes to `Detection2DArray`, aligned compressed/raw depth, camera intrinsics, and TF; it publishes `Detection3DArray`, RViz markers, and optional frustum clouds. It buffers depth frames and pairs each detection message with the nearest acceptable depth timestamp rather than using strict synchronization. TF transforms decoded poses from the camera optical frame to `target_frame` (`map` by default); failed lookups deliberately retain camera-frame output.
Paths: `rgbd3d/ros2_node.py`, `rgbd3d/box_to_ros.py`, `rgbd3d/config.py`.

#### Detection admission and duplicate control
Incoming 2D hypotheses are normalized to class names/IDs, filtered by confidence and the supported `car`/`person` allow-list, then reduced with greedy 2D suppression. Suppression considers conventional IoU and containment coverage, which handles nested detectNet boxes. After 3D inference, a ground-plane proximity check suppresses duplicate boxes that represent the same object.
Paths: `rgbd3d/ros2_node.py`, `rgbd3d/dedupe.py`, `rgbd3d/config.py`.

#### Depth-to-frustum geometry
For every retained 2D ROI, the pipeline validates depth values, clamps image bounds, deprojects valid pixels using calibrated pinhole intrinsics, and rejects sparse clouds. It estimates an object-specific scale from apparent image height and median depth, then rotates the frustum toward the network’s canonical forward view. This is the key boundary between camera/depth geometry and model-ready point clouds.
Paths: `rgbd3d/frustum.py`, `rgbd3d/ros2_node.py`, `rgbd3d/fp_constants.py`.

#### Two-stage Frustum-PointNet inference
`FrustumPointNet3D` runs a split Frustum-PointNet v1 pipeline: stage A produces point segmentation logits; host-side NumPy masking selects and samples foreground points; stage B regresses center, heading bins/residuals, and size clusters/residuals. TensorRT is the intended accelerated backend, with lazy engine setup/buffer management; ONNX Runtime provides a portable offline fallback. The split is architectural because the original masking operation cannot be exported to ONNX.
Paths: `rgbd3d/model3d.py`, `rgbd3d/box_decode.py`, `tools/export_frustum_pointnet.py`.

#### Metric-box reconstruction and outputs
Decoded model outputs are re-centered on the segmented foreground centroid, unrotated from canonical frustum coordinates, and divided by the per-object scale to recover camera-frame metres. Size and orientation come from KITTI-trained model priors rather than geometric extents/PCA; class-to-size-cluster selection is therefore a material accuracy constraint for model-scale vehicles. The ROS layer converts this result into pose/size messages, transforms pose only across frames, and emits markers; dense raw frustum clouds are generated only if subscribed.
Paths: `rgbd3d/box_decode.py`, `rgbd3d/box_to_ros.py`, `rgbd3d/visualize.py`.

#### Runtime dependencies and operational constraints
The deployment graph requires a RealSense source with depth aligned to color, an external detectNet ROS graph, and TF from camera to robot/localization map. TensorRT/CUDA is the performance path; ONNX Runtime supports offline validation. The node uses BEST_EFFORT depth/detection QoS by default to avoid Wi-Fi retransmission backlog and pins BLAS/OpenMP threading to avoid starving the single-threaded ROS executor.
Paths: `rgbd3d/ros2_node.py`, `rgbd3d/model3d.py`, `package.xml`.

Main entry points: [`rgbd3d/frustum.py`](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/frustum.py) (ROI frustum builder), [`rgbd3d/ros2_node.py`](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/ros2_node.py) (Localizer3D node), [`rgbd3d/box_to_ros.py`](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/box_to_ros.py) (ROS box conversion), [`rgbd3d/box_decode.py`](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/box_decode.py) (Metric box decoder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Upstream ROS inputs"]
  node_detectnet["detectNet detections<br/>external ROS graph"]
  node_realsense["Aligned RealSense depth<br/>camera source"]
  node_camera_info["Camera intrinsics<br/>ROS input"]
  node_tf["TF frame graph<br/>ROS transform service"]
end

subgraph group_runtime["ROS localization runtime"]
  node_localizer{{"Localizer3D node<br/>ROS node<br/>[ros2_node.py]"}}
  node_admission["Detection admission<br/>filter and 2D dedupe<br/>[dedupe.py]"]
  node_depth_pairing["Depth timestamp pairing<br/>depth buffer<br/>[ros2_node.py]"]
  node_ground_dedupe["Ground-plane dedupe<br/>3D duplicate filter<br/>[dedupe.py]"]
end

subgraph group_geometry["Depth geometry"]
  node_frustum["ROI frustum builder<br/>depth deprojection<br/>[frustum.py]"]
  node_decoder["Metric box decoder<br/>3D reconstruction<br/>[box_decode.py]"]
end

subgraph group_inference["Frustum-PointNet inference"]
  node_model{{"FrustumPointNet3D<br/>split 3D model<br/>[model3d.py]"}}
  node_segmentation["Stage A segmentation<br/>point logits<br/>[model3d.py]"]
  node_foreground["Host foreground masking<br/>NumPy point selection<br/>[model3d.py]"]
  node_box_regression["Stage B box regression<br/>center heading size prediction<br/>[model3d.py]"]
  node_backends["TensorRT / ONNX Runtime<br/>inference backend<br/>[model3d.py]"]
end

subgraph group_outputs["ROS outputs"]
  node_ros_conversion["ROS box conversion<br/>message and transform adapter<br/>[box_to_ros.py]"]
  node_detections3d["Detection3DArray<br/>ROS publication<br/>[ros2_node.py]"]
  node_visualization["Markers and frustum clouds<br/>ROS visualization<br/>[visualize.py]"]
end

node_detectnet -->|"Detection2DArray"| node_localizer
node_realsense -->|"compressed/raw depth"| node_localizer
node_camera_info -->|"intrinsics"| node_localizer
node_localizer -->|"2D hypotheses"| node_admission
node_localizer -->|"buffer frames"| node_depth_pairing
node_admission -->|"retained ROIs"| node_frustum
node_depth_pairing -->|"nearest aligned depth"| node_frustum
node_camera_info -->|"pinhole calibration"| node_frustum
node_frustum -->|"canonical points and scale"| node_model
node_model -->|"stage A"| node_segmentation
node_segmentation -->|"point logits"| node_foreground
node_foreground -->|"sampled foreground"| node_box_regression
node_box_regression -->|"box predictions"| node_decoder
node_frustum -->|"centroid, rotation, scale"| node_decoder
node_decoder -->|"camera-frame metric boxes"| node_ground_dedupe
node_ground_dedupe -->|"unique boxes"| node_ros_conversion
node_tf -->|"camera to target transform"| node_ros_conversion
node_ros_conversion -->|"poses and sizes"| node_detections3d
node_ros_conversion -->|"boxes and poses"| node_visualization
node_frustum -.->|"optional raw clouds"| node_visualization
node_backends -->|"execution provider"| node_model

click node_localizer "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/ros2_node.py"
click node_admission "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/dedupe.py"
click node_depth_pairing "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/ros2_node.py"
click node_frustum "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/frustum.py"
click node_model "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/model3d.py"
click node_segmentation "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/model3d.py"
click node_foreground "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/model3d.py"
click node_box_regression "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/model3d.py"
click node_decoder "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/box_decode.py"
click node_ground_dedupe "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/dedupe.py"
click node_ros_conversion "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/box_to_ros.py"
click node_detections3d "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/ros2_node.py"
click node_visualization "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/visualize.py"
click node_backends "https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/model3d.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_detectnet,node_realsense,node_camera_info,node_tf toneBlue
class node_localizer,node_admission,node_depth_pairing,node_ground_dedupe toneAmber
class node_frustum,node_decoder toneMint
class node_model,node_segmentation,node_foreground,node_box_regression,node_backends toneRose
class node_ros_conversion,node_detections3d,node_visualization toneIndigo
```

## Components

### Upstream ROS inputs

Systems owned outside this package.

- **detectNet detections**: external ROS graph · Provides 2D hypotheses only.
- **Aligned RealSense depth**: camera source · Depth must be aligned to color.
- **Camera intrinsics**: ROS input
- **TF frame graph**: ROS transform service · Supplies camera-to-target transforms.

### ROS localization runtime

- [Localizer3D node](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/ros2_node.py): ROS node · Buffers nearest depth frames and orchestrates localization. · `rgbd3d/ros2_node.py`
- [Detection admission](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/dedupe.py): filter and 2D dedupe · Normalizes classes, applies allow-list/confidence gates, and suppresses nested boxes. · `rgbd3d/dedupe.py`
- [Depth timestamp pairing](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/ros2_node.py): depth buffer · Selects the nearest acceptable depth frame instead of strict synchronization. · `rgbd3d/ros2_node.py`
- [Ground-plane dedupe](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/dedupe.py): 3D duplicate filter · Suppresses nearby boxes representing the same object. · `rgbd3d/dedupe.py`

### Depth geometry

- [ROI frustum builder](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/frustum.py): depth deprojection · Validates depth, deprojects ROI pixels, estimates scale, and canonicalizes the frustum. · `rgbd3d/frustum.py`
- [Metric box decoder](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/box_decode.py): 3D reconstruction · Recenters, unrotates, rescales, and selects class-constrained size priors. · `rgbd3d/box_decode.py`

### Frustum-PointNet inference

- [FrustumPointNet3D](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/model3d.py): split 3D model · Runs segmentation then box regression. · `rgbd3d/model3d.py`
- [Stage A segmentation](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/model3d.py): point logits · `rgbd3d/model3d.py`
- [Host foreground masking](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/model3d.py): NumPy point selection · Masks and samples foreground points outside exported ONNX graph. · `rgbd3d/model3d.py`
- [Stage B box regression](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/model3d.py): center heading size prediction · Predicts center, heading bins/residuals, and KITTI size clusters/residuals. · `rgbd3d/model3d.py`
- [TensorRT / ONNX Runtime](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/model3d.py): inference backend · TensorRT is the accelerated path; ONNX Runtime supports portable offline validation. · `rgbd3d/model3d.py`

### ROS outputs

- [ROS box conversion](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/box_to_ros.py): message and transform adapter · Transforms pose to target frame when TF is available; size remains camera-derived. · `rgbd3d/box_to_ros.py`
- [Detection3DArray](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/ros2_node.py): ROS publication · Camera-frame output is retained when TF lookup fails. · `rgbd3d/ros2_node.py`
- [Markers and frustum clouds](https://github.com/raidhu/object_projection_3d/blob/main/rgbd3d/visualize.py): ROS visualization · Dense frustum clouds are emitted only when subscribed. · `rgbd3d/visualize.py`

## Connections

- detectNet detections → Localizer3D node: Detection2DArray
- Aligned RealSense depth → Localizer3D node: compressed/raw depth
- Camera intrinsics → Localizer3D node: intrinsics
- Localizer3D node → Detection admission: 2D hypotheses
- Localizer3D node → Depth timestamp pairing: buffer frames
- Detection admission → ROI frustum builder: retained ROIs
- Depth timestamp pairing → ROI frustum builder: nearest aligned depth
- Camera intrinsics → ROI frustum builder: pinhole calibration
- ROI frustum builder → FrustumPointNet3D: canonical points and scale
- FrustumPointNet3D → Stage A segmentation: stage A
- Stage A segmentation → Host foreground masking: point logits
- Host foreground masking → Stage B box regression: sampled foreground
- Stage B box regression → Metric box decoder: box predictions
- ROI frustum builder → Metric box decoder: centroid, rotation, scale
- Metric box decoder → Ground-plane dedupe: camera-frame metric boxes
- Ground-plane dedupe → ROS box conversion: unique boxes
- TF frame graph → ROS box conversion: camera to target transform
- ROS box conversion → Detection3DArray: poses and sizes
- ROS box conversion → Markers and frustum clouds: boxes and poses
- ROI frustum builder → Markers and frustum clouds: optional raw clouds
- TensorRT / ONNX Runtime → FrustumPointNet3D: execution provider