How rishavs-ymsli/robotic-arm-with-vision works
1. What this repo is
- tray_pick is a ROS 2 Humble vision-guided pick-and-place pipeline for a UR10e with a Robotiq gripper.
- The system is split into a perception side, a motion/control side, and small calibration/debug tools.
2. Top-level data flow
- External camera topics provide synchronized RGB, depth, and camera info.
- The vision pipeline detects tray objects and publishes grasp poses.
- The pick pipeline consumes those poses, plans arm motion through MoveIt 2, and commands the gripper via a ROS topic bridge.
3. Configuration boundary
- config/settings.py is the single source of tunable parameters.
- It holds calibration mode, tray geometry, warp points, ready/drop poses, gripper limits, and debug display flags.
- This keeps geometry and runtime assumptions out of the operational code.
4. Perception subsystem
- camera/stream.py handles synchronized RGB+depth input and image warping/registration around the tray view.
- camera/detector.py performs depth-based object detection using contours/blobs and baseline subtraction.
- camera/transform.py converts image-space detections into robot-space XY coordinates using either homography or pinhole/extrinsic-style mapping.
5. Vision node boundary
- vision_node.py is the bridge from raw camera data to robot-consumable perception outputs.
- It publishes /tray/objects as PoseArray in base_link, plus visualization artifacts like /tray/detection_vis and /tray/object_mask.
- This node is the main handoff point between sensing and manipulation.
6. Manipulation subsystem
- robot/arm.py encapsulates MoveIt 2 motion control for the UR10e, including planning and Cartesian moves.
- robot/gripper.py wraps the Isaac Sim gripper bridge on /gripper_command.
- robot/scene.py defines the MoveIt planning scene objects such as tray, walls, and floor so collision avoidance matches the workspace.
7. Pick-and-place execution
- pick_node.py implements the state machine that receives object poses, moves the arm to grasp them, and then moves to the drop location.
- It is the consumer of perception output and the coordinator of robot actions.
- The system’s action boundary is therefore “perception publishes candidates, pick node decides and executes.”
8. System orchestration
- main.py runs the vision and pick components together for the full pipeline.
- This is the simplest path from camera input to autonomous picking, assuming the ROS infrastructure is already running.
9. Calibration and validation tools
- calibrate_corners.py is an interactive workflow to record tray corner TCP positions for homography-based calibration.
- verify_calibration.py moves the arm to those points for sanity checking.
- camera/transform.py and the calibration scripts together define the mapping from tray image geometry to robot coordinates.
10. Manual/debug paths
- cartesian_keyboard_teleop.py provides direct TCP jogging for manual testing and recovery.
- vision_node.py also supports keyboard interaction for baseline capture, reset, and corner inspection.
- SHOW_DEBUG_WINDOWS controls whether the perception loop exposes OpenCV windows, which matters for headless deployment.
11. External runtime dependencies
- The repo assumes ROS topics from a RealSense-style camera, a running MoveIt 2 stack for the UR10e, and an Isaac Sim gripper bridge.
- The repo itself is not the full robot stack; it plugs into those services and exchanges data purely through ROS interfaces.
Main entry points: settings.py Settings · pick_node.py Pick node · arm.py Arm · vision_node.py Vision node
Components 17
Perception
Camera ingestion, detection, and coordinate transforms.
- Camera stream · Synchronizes RGB/depth input and handles tray-view warping.
camera/stream.py - Detector · Depth-based object detection using contours and baseline subtraction.
camera/detector.py - Transform · Maps image detections into robot-space coordinates.
camera/transform.py - YOLO model · Packaged detector weights used by the vision pipeline.
yolov8n.pt
Manipulation
Robot motion, gripper control, and planning scene.
Orchestration
Top-level runtime and pick execution.
- Vision node · Bridges camera data to robot-consumable perception outputs.
vision_node.py - Pick node · State machine that grasps detected objects and executes drops.
pick_node.py - Main · Runs vision and pick components together.
main.py
Tools
Calibration, validation, and manual debug utilities.
- Calibrate corners · Interactive recording of tray corner TCP positions.
calibrate_corners.py - Verify calibration · Moves the arm to calibration points for sanity checks.
verify_calibration.py - Teleop · Manual TCP jogging for testing and recovery.
cartesian_keyboard_teleop.py
Config
Shared tunable parameters and calibration assumptions.
- Settings · Central runtime and calibration parameters.
config/settings.py
Other components
- Camera topics · External synchronized RGB, depth, and camera info streams.
- MoveIt stack · Robot planning and execution backend for the UR10e.
- Gripper bridge · Isaac Sim or equivalent bridge consuming gripper commands.
Connections 22
- Camera topics → Camera stream: rgb/depth/info · Provides synchronized camera inputs.
- Settings → Camera stream: warp/capture · Supplies tray-view and debug settings.
- Settings → Detector: thresholds · Supplies detection and baseline parameters.
- Settings → Transform: calibration · Supplies geometry and mapping parameters.
- YOLO model → Detector: weights · Detector model artifact used by perception.
- Camera stream → Detector: frames · Feeds registered images into detection.
- Detector → Transform: detections · Passes image-space object candidates.
- Transform → Vision node: poses · Produces robot-frame grasp candidates.
- Vision node → Pick node: pose array · Publishes object candidates for picking.
- Vision node → Main: vision output · Provides the sensing side of the full pipeline.
- Pick node → Arm: motion goals · Sends grasp, retreat, and drop motions.
- Pick node → Gripper: open/close · Commands gripper actuation during picks.
- Scene → Arm: constraints · Defines collision geometry for planning.
- MoveIt stack → Arm: planning backend · Executes the motion planning and arm control path.
- Gripper bridge → Gripper: command channel · Receives gripper commands over ROS.
- Settings → Pick node: poses/limits · Supplies ready, drop, and gripper parameters.
- Settings → Scene: workspace dims · Defines planning scene geometry.
- Calibrate corners → Settings: corner data · Produces tray calibration inputs.
- Verify calibration → Arm: test moves · Uses the arm to validate calibration points.
- Teleop → Arm: manual jog · Direct manual control of the TCP.
- Main → Pick node: start · Launches the manipulation coordinator.
- Main → Vision node: start · Launches the perception bridge.