How sahilrajpurkar03/nlp-pnp-robotic-arm works
Project Type and Purpose
• SPARC is a full-stack robotics simulation and control system. It integrates a physics/graphics simulator (NVIDIA Isaac Sim via a USD scene), a ROS2 Humble middleware layer, a perception stack (YOLOv8‐OBB), a motion planning stack (MoveIt + ROS2 Control), and a natural‐language chatbot interface (Ollama LLM).
• The goal is to let users type English commands and see a Franka or UR5 arm pick and place objects in simulation.
File Structure Analysis
Top-level logical components:
a. pick_place_chatbot_ui/
• Scripts that run the UI, launch the chatbot, bridge between LLM and ROS2 topics.
b. src/robot_description/
• URDF/XACRO robot model, meshes, USD exports—defines the robot’s kinematic and collision geometry.
c. src/ur5_moveit_config/
• MoveIt configuration: SRDF, planning parameters, controllers, launch files, sample C++ pick-and-place executables.
d. src/yolov8_obb_msgs/
• Custom ROS2 message definitions (InferenceResult, PickPlaceStatus, Yolov8Inference) for the perception → planning pipeline.
e. src/yolov8obb_object_detection/
• A ROS2 node package wrapping Ultralytics YOLOv8‐OBB: data collection, publisher, subscriber nodes, training data.
f. recording_scripts/
• Utility scripts for offline data capture.
g. SPARC.usd + docs/PickPlace.gif
• Simulation scene and demo assets.
h. requirements.txt, launch scripts, README.md
Architecture Overview
Main components to represent:
A. Simulation Engine (Isaac Sim)
– Loads SPARC.usd, provides camera sensor output (ROS2 image topics), physics.
B. Perception Node (yolov8obb_object_detection)
C. Message Definitions (yolov8_obb_msgs)
– Defines the ROS2 topics and message types used by perception and downstream nodes.
D. Target Publisher (target_publisher.py in pick_place_chatbot_ui)
– Receives selected object or coordinates, publishes a /target_point topic.
E. Planner & Controller (MoveIt + ros2_control)
– ur5_moveit_config package. Launch files spin up move_group, RViz, controllers.
– Subscribes to /target_point, plans trajectories, sends commands to robot controllers (position_trajectory_controller, gripper).
F. Chatbot UI (pick_place_chatbot_ui)
– cmd_bridge.py and llm_mapper.py call the LLM, interpret user text, select object class or coordinates, and forward commands via ROS2.
G. ROS2 Core
– master node (ros2 daemon), parameter server, topic bus.
H. (Optional) Data Recorder Scripts
Interactions and Data Flow
• Isaac Sim publishes /camera/image_raw → YOLO node
• YOLO node publishes /yolo_obb_inference → Chatbot mapper or direct subscriber
• LLM UI captures user text → cmd_bridge → llm_mapper → publishes /target_point
• MoveIt’s move_group subscribes to /target_point → generates trajectory → controllers → robot (simulated joint states)
• Feedback (PickPlaceStatus) published back to UI for status updates
Architectural Patterns & Principles
• ROS2 Publisher/Subscriber for loose coupling
• Separate perception, planning, and interface modules
• Reusable URDF/XACRO and MoveIt config packages
• Clear message definitions package for cross‐node contracts
• Modular Python scripts for UI bridging
Final Checklist
• All major components clearly boxed and named.
• Call out key packages: robot_description, ur5_moveit_config, yolov8obb_object_detection, pick_place_chatbot_ui.
• External services: Isaac Sim, Ollama LLM.
• You capture the flow: user → LLM → ROS2 → sim → ROS2 → planning → sim feedback → UI.