# rishavs-ymsli/robotic-arm-with-vision architecture

> Architecture diagram of the GitHub repository rishavs-ymsli/robotic-arm-with-vision, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rishavs-ymsli/robotic-arm-with-vision
- Repository: https://github.com/rishavs-ymsli/robotic-arm-with-vision

## Overview

1. **What this repo is**
   - `tray_pick` is a ROS 2 Humble vision-guided pick-and-place pipeline for a UR10e with a Robotiq gripper.
   - The system is split into a perception side, a motion/control side, and small calibration/debug tools.

2. **Top-level data flow**
   - External camera topics provide synchronized RGB, depth, and camera info.
   - The vision pipeline detects tray objects and publishes grasp poses.
   - The pick pipeline consumes those poses, plans arm motion through MoveIt 2, and commands the gripper via a ROS topic bridge.

3. **Configuration boundary**
   - `config/settings.py` is the single source of tunable parameters.
   - It holds calibration mode, tray geometry, warp points, ready/drop poses, gripper limits, and debug display flags.
   - This keeps geometry and runtime assumptions out of the operational code.

4. **Perception subsystem**
   - `camera/stream.py` handles synchronized RGB+depth input and image warping/registration around the tray view.
   - `camera/detector.py` performs depth-based object detection using contours/blobs and baseline subtraction.
   - `camera/transform.py` converts image-space detections into robot-space XY coordinates using either homography or pinhole/extrinsic-style mapping.

5. **Vision node boundary**
   - `vision_node.py` is the bridge from raw camera data to robot-consumable perception outputs.
   - It publishes `/tray/objects` as `PoseArray` in `base_link`, plus visualization artifacts like `/tray/detection_vis` and `/tray/object_mask`.
   - This node is the main handoff point between sensing and manipulation.

6. **Manipulation subsystem**
   - `robot/arm.py` encapsulates MoveIt 2 motion control for the UR10e, including planning and Cartesian moves.
   - `robot/gripper.py` wraps the Isaac Sim gripper bridge on `/gripper_command`.
   - `robot/scene.py` defines the MoveIt planning scene objects such as tray, walls, and floor so collision avoidance matches the workspace.

7. **Pick-and-place execution**
   - `pick_node.py` implements the state machine that receives object poses, moves the arm to grasp them, and then moves to the drop location.
   - It is the consumer of perception output and the coordinator of robot actions.
   - The system’s action boundary is therefore “perception publishes candidates, pick node decides and executes.”

8. **System orchestration**
   - `main.py` runs the vision and pick components together for the full pipeline.
   - This is the simplest path from camera input to autonomous picking, assuming the ROS infrastructure is already running.

9. **Calibration and validation tools**
   - `calibrate_corners.py` is an interactive workflow to record tray corner TCP positions for homography-based calibration.
   - `verify_calibration.py` moves the arm to those points for sanity checking.
   - `camera/transform.py` and the calibration scripts together define the mapping from tray image geometry to robot coordinates.

10. **Manual/debug paths**
   - `cartesian_keyboard_teleop.py` provides direct TCP jogging for manual testing and recovery.
   - `vision_node.py` also supports keyboard interaction for baseline capture, reset, and corner inspection.
   - `SHOW_DEBUG_WINDOWS` controls whether the perception loop exposes OpenCV windows, which matters for headless deployment.

11. **External runtime dependencies**
   - The repo assumes ROS topics from a RealSense-style camera, a running MoveIt 2 stack for the UR10e, and an Isaac Sim gripper bridge.
   - The repo itself is not the full robot stack; it plugs into those services and exchanges data purely through ROS interfaces.

Main entry points: [`config/settings.py`](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/config/settings.py) (Settings), [`pick_node.py`](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/pick_node.py) (Pick node), [`robot/arm.py`](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/robot/arm.py) (Arm), [`vision_node.py`](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/vision_node.py) (Vision node)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_perception["Perception"]
  node_camera_stream["Camera stream<br/>vision input<br/>[stream.py]"]
  node_camera_detector["Detector<br/>vision<br/>[detector.py]"]
  node_camera_transform["Transform<br/>geometry<br/>[transform.py]"]
  node_yolov8_model["YOLO model<br/>weights<br/>[yolov8n.pt]"]
end

subgraph group_manipulation["Manipulation"]
  node_robot_arm["Arm<br/>moveit control<br/>[arm.py]"]
  node_robot_gripper["Gripper<br/>topic bridge<br/>[gripper.py]"]
  node_robot_scene[("Scene<br/>[scene.py]")]
end

subgraph group_orchestration["Orchestration"]
  node_vision_node(("Vision node<br/>ros node<br/>[vision_node.py]"))
  node_pick_node(("Pick node<br/>ros node<br/>[pick_node.py]"))
  node_main(("Main<br/>entrypoint<br/>[main.py]"))
end

subgraph group_tools["Tools"]
  node_calibrate_corners(("Calibrate corners<br/>tool"))
  node_verify_calibration(("Verify calibration<br/>tool"))
  node_teleop(("Teleop<br/>tool"))
end

subgraph group_config["Config"]
  node_config_settings["Settings<br/>params<br/>[settings.py]"]
end

node_external_camera["Camera topics<br/>ros input"]
node_external_moveit[("MoveIt stack<br/>external runtime")]
node_external_gripper_bridge["Gripper bridge<br/>external runtime"]

node_external_camera -->|"rgb/depth/info"| node_camera_stream
node_config_settings -->|"warp/capture"| node_camera_stream
node_config_settings -->|"thresholds"| node_camera_detector
node_config_settings -->|"calibration"| node_camera_transform
node_yolov8_model -->|"weights"| node_camera_detector
node_camera_stream -->|"frames"| node_camera_detector
node_camera_detector -->|"detections"| node_camera_transform
node_camera_transform -->|"poses"| node_vision_node
node_vision_node -->|"pose array"| node_pick_node
node_vision_node -->|"vision output"| node_main
node_pick_node -->|"motion goals"| node_robot_arm
node_pick_node -->|"open/close"| node_robot_gripper
node_robot_scene -->|"constraints"| node_robot_arm
node_external_moveit -->|"planning backend"| node_robot_arm
node_external_gripper_bridge -->|"command channel"| node_robot_gripper
node_config_settings -->|"poses/limits"| node_pick_node
node_config_settings -->|"workspace dims"| node_robot_scene
node_calibrate_corners -->|"corner data"| node_config_settings
node_verify_calibration -->|"test moves"| node_robot_arm
node_teleop -->|"manual jog"| node_robot_arm
node_main -->|"start"| node_pick_node
node_main -->|"start"| node_vision_node

click node_config_settings "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/config/settings.py"
click node_camera_stream "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/camera/stream.py"
click node_camera_detector "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/camera/detector.py"
click node_camera_transform "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/camera/transform.py"
click node_vision_node "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/vision_node.py"
click node_pick_node "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/pick_node.py"
click node_main "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/main.py"
click node_robot_arm "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/robot/arm.py"
click node_robot_gripper "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/robot/gripper.py"
click node_robot_scene "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/robot/scene.py"
click node_calibrate_corners "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/calibrate_corners.py"
click node_verify_calibration "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/verify_calibration.py"
click node_teleop "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/cartesian_keyboard_teleop.py"
click node_yolov8_model "https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/yolov8n.pt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_camera_stream,node_camera_detector,node_camera_transform,node_yolov8_model toneBlue
class node_robot_arm,node_robot_gripper,node_robot_scene toneAmber
class node_vision_node,node_pick_node,node_main toneMint
class node_calibrate_corners,node_verify_calibration,node_teleop toneRose
class node_config_settings toneIndigo
class node_external_camera,node_external_moveit,node_external_gripper_bridge toneNeutral
```

## Components

### Perception

Camera ingestion, detection, and coordinate transforms.

- [Camera stream](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/camera/stream.py): vision input · Synchronizes RGB/depth input and handles tray-view warping. · `camera/stream.py`
- [Detector](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/camera/detector.py): vision · Depth-based object detection using contours and baseline subtraction. · `camera/detector.py`
- [Transform](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/camera/transform.py): geometry · Maps image detections into robot-space coordinates. · `camera/transform.py`
- [YOLO model](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/yolov8n.pt): weights · Packaged detector weights used by the vision pipeline. · `yolov8n.pt`

### Manipulation

Robot motion, gripper control, and planning scene.

- [Arm](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/robot/arm.py): moveit control · Encapsulates UR10e motion planning and Cartesian moves. · `robot/arm.py`
- [Gripper](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/robot/gripper.py): topic bridge · Wraps gripper commands sent over ROS. · `robot/gripper.py`
- [Scene](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/robot/scene.py): planning scene · Defines tray, walls, and floor collision objects. · `robot/scene.py`

### Orchestration

Top-level runtime and pick execution.

- [Vision node](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/vision_node.py): ros node · Bridges camera data to robot-consumable perception outputs. · `vision_node.py`
- [Pick node](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/pick_node.py): ros node · State machine that grasps detected objects and executes drops. · `pick_node.py`
- [Main](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/main.py): entrypoint · Runs vision and pick components together. · `main.py`

### Tools

Calibration, validation, and manual debug utilities.

- [Calibrate corners](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/calibrate_corners.py): tool · Interactive recording of tray corner TCP positions. · `calibrate_corners.py`
- [Verify calibration](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/verify_calibration.py): tool · Moves the arm to calibration points for sanity checks. · `verify_calibration.py`
- [Teleop](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/cartesian_keyboard_teleop.py): tool · Manual TCP jogging for testing and recovery. · `cartesian_keyboard_teleop.py`

### Config

Shared tunable parameters and calibration assumptions.

- [Settings](https://github.com/rishavs-ymsli/robotic-arm-with-vision/blob/master/config/settings.py): params · Central runtime and calibration parameters. · `config/settings.py`

### Other components

- **Camera topics**: ros input · External synchronized RGB, depth, and camera info streams.
- **MoveIt stack**: external runtime · Robot planning and execution backend for the UR10e.
- **Gripper bridge**: external runtime · Isaac Sim or equivalent bridge consuming gripper commands.

## Connections

- Camera topics → Camera stream: rgb/depth/info · Provides synchronized camera inputs.
- Settings → Camera stream: warp/capture · Supplies tray-view and debug settings.
- Settings → Detector: thresholds · Supplies detection and baseline parameters.
- Settings → Transform: calibration · Supplies geometry and mapping parameters.
- YOLO model → Detector: weights · Detector model artifact used by perception.
- Camera stream → Detector: frames · Feeds registered images into detection.
- Detector → Transform: detections · Passes image-space object candidates.
- Transform → Vision node: poses · Produces robot-frame grasp candidates.
- Vision node → Pick node: pose array · Publishes object candidates for picking.
- Vision node → Main: vision output · Provides the sensing side of the full pipeline.
- Pick node → Arm: motion goals · Sends grasp, retreat, and drop motions.
- Pick node → Gripper: open/close · Commands gripper actuation during picks.
- Scene → Arm: constraints · Defines collision geometry for planning.
- MoveIt stack → Arm: planning backend · Executes the motion planning and arm control path.
- Gripper bridge → Gripper: command channel · Receives gripper commands over ROS.
- Settings → Pick node: poses/limits · Supplies ready, drop, and gripper parameters.
- Settings → Scene: workspace dims · Defines planning scene geometry.
- Calibrate corners → Settings: corner data · Produces tray calibration inputs.
- Verify calibration → Arm: test moves · Uses the arm to validate calibration points.
- Teleop → Arm: manual jog · Direct manual control of the TCP.
- Main → Pick node: start · Launches the manipulation coordinator.
- Main → Vision node: start · Launches the perception bridge.
