# sahilrajpurkar03/nlp-pnp-robotic-arm architecture

> Architecture diagram of the GitHub repository sahilrajpurkar03/nlp-pnp-robotic-arm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sahilrajpurkar03/nlp-pnp-robotic-arm
- Repository: https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm

## Overview

##### Project Type and Purpose
- SPARC is a full-stack robotics simulation and control system. It integrates a physics/graphics simulator (NVIDIA Isaac Sim via a USD scene), a ROS2 Humble middleware layer, a perception stack (YOLOv8‐OBB), a motion planning stack (MoveIt + ROS2 Control), and a natural‐language chatbot interface (Ollama LLM).
- The goal is to let users type English commands and see a Franka or UR5 arm pick and place objects in simulation.

##### File Structure Analysis
Top-level logical components:
a. pick_place_chatbot_ui/
- Scripts that run the UI, launch the chatbot, bridge between LLM and ROS2 topics.
b. src/robot_description/
- URDF/XACRO robot model, meshes, USD exports—defines the robot’s kinematic and collision geometry.
c. src/ur5_moveit_config/
- MoveIt configuration: SRDF, planning parameters, controllers, launch files, sample C++ pick-and-place executables.
d. src/yolov8_obb_msgs/
- Custom ROS2 message definitions (InferenceResult, PickPlaceStatus, Yolov8Inference) for the perception → planning pipeline.
e. src/yolov8obb_object_detection/
- A ROS2 node package wrapping Ultralytics YOLOv8‐OBB: data collection, publisher, subscriber nodes, training data.
f. recording_scripts/
- Utility scripts for offline data capture.
g. SPARC.usd + docs/PickPlace.gif
- Simulation scene and demo assets.
h. requirements.txt, launch scripts, README.md

##### Architecture Overview
Main components to represent:
A. Simulation Engine (Isaac Sim)
– Loads SPARC.usd, provides camera sensor output (ROS2 image topics), physics.
B. Perception Node (yolov8obb_object_detection)
C. Message Definitions (yolov8_obb_msgs)
– Defines the ROS2 topics and message types used by perception and downstream nodes.
D. Target Publisher (target_publisher.py in pick_place_chatbot_ui)
– Receives selected object or coordinates, publishes a /target_point topic.
E. Planner & Controller (MoveIt + ros2_control)
– ur5_moveit_config package. Launch files spin up move_group, RViz, controllers.
– Subscribes to /target_point, plans trajectories, sends commands to robot controllers (position_trajectory_controller, gripper).
F. Chatbot UI (pick_place_chatbot_ui)
– cmd_bridge.py and llm_mapper.py call the LLM, interpret user text, select object class or coordinates, and forward commands via ROS2.
G. ROS2 Core
– master node (ros2 daemon), parameter server, topic bus.
H. (Optional) Data Recorder Scripts

##### Interactions and Data Flow
- Isaac Sim publishes /camera/image_raw → YOLO node
- YOLO node publishes /yolo_obb_inference → Chatbot mapper or direct subscriber
- LLM UI captures user text → cmd_bridge → llm_mapper → publishes /target_point
- MoveIt’s move_group subscribes to /target_point → generates trajectory → controllers → robot (simulated joint states)
- Feedback (PickPlaceStatus) published back to UI for status updates

##### Architectural Patterns & Principles
- ROS2 Publisher/Subscriber for loose coupling
- Separate perception, planning, and interface modules
- Reusable URDF/XACRO and MoveIt config packages
- Clear message definitions package for cross‐node contracts
- Modular Python scripts for UI bridging

##### Final Checklist
- All major components clearly boxed and named.
- Call out key packages: robot_description, ur5_moveit_config, yolov8obb_object_detection, pick_place_chatbot_ui.
- External services: Isaac Sim, Ollama LLM.
- You capture the flow: user → LLM → ROS2 → sim → ROS2 → planning → sim feedback → UI.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% ROS2 Core
    subgraph "ROS2 Core"
        ROS2_DAEMON["ROS2 Daemon\n(DDS)"]:::core
        PARAM_SERVER["Parameter Server"]:::core
        TOPIC_BUS["Topic Bus"]:::core
    end

    %% Simulation Engine
    subgraph "Simulation Engine"  
        ISSIM["Isaac Sim"]:::sim
        USD_SCENE["SPARC.usd"]:::sim
        CAMERA_XACRO["camera.xacro"]:::sim
    end

    %% Perception
    subgraph "Perception Node"  
        YOLO_PUB["yolov8_obb_publisher"]:::perception
        YOLO_SUB["yolov8_obb_subscriber"]:::perception
        YOLO_LAUNCH["yolov8_obb_launch"]:::perception
    end

    %% Message Definitions
    subgraph "Message Definitions"  
        INFERENCE_RESULT["InferenceResult.msg"]:::msgs
        YOLO_INFERENCE_MSG["Yolov8Inference.msg"]:::msgs
        PICKPLACE_STATUS["PickPlaceStatus.msg"]:::msgs
    end

    %% Planning & Control
    subgraph "Planning & Controller"  
        MOVE_GROUP["move_group (ROS2)"]:::planning
        RVIZ["RViz"]:::planning
        SPAWN_CTRL["spawn_controllers.launch"]:::planning
        DEMO_LAUNCH["demo.launch"]:::planning
        UR5_XACRO["ur5.urdf.xacro"]:::planning
        SRDF["ur5.srdf"]:::planning
        ROS2_CONTROL_CFG["moveit_controllers.yaml"]:::planning
        CPP_PICK_R["ur5_pick_place_cpp_r.cpp"]:::planning
        CPP_PICK_S["ur5_pick_place_cpp_s.cpp"]:::planning
    end

    %% Chatbot UI
    subgraph "Chatbot UI"
        CMD_BRIDGE["cmd_bridge.py"]:::ui
        LLM_MAPPER["llm_mapper.py"]:::ui
        TARGET_PUB["target_publisher.py"]:::ui
        INDEX_HTML["index.html"]:::ui
        LAUNCH_UI["launch.sh"]:::ui
    end

    %% Robot Description
    subgraph "Robot Description"
        UR5_URDF["ur5.urdf.xacro"]:::desc
        GRIPPER_XACRO["robotiq_85_gripper.xacro"]:::desc
        UR5_MESH["meshes/ur5"]:::desc
        GRIPPER_MESH["robotiq_meshes"]:::desc
    end

    %% Data Recorder
    subgraph "Data Recorder Scripts"
        REC_ENV["record_env.py"]:::rec
        REC_MOV_CAM["record_mov_cam.py"]:::rec
        REC_YOLO["record_yolo_det.py"]:::rec
    end

    %% Topics
    IMAGE_TOPIC["/camera/image_raw"]:::topic
    YOLO_TOPIC["/yolo_obb_inference"]:::topic
    TARGET_TOPIC["/target_point"]:::topic
    STATUS_TOPIC["/pick_place_status"]:::topic

    %% Interactions
    ISSIM -->|Publishes| IMAGE_TOPIC
    IMAGE_TOPIC -->|Subscribes| YOLO_SUB
    YOLO_SUB -->|Runs Inference| YOLO_PUB
    YOLO_PUB -->|Publishes| YOLO_TOPIC
    YOLO_TOPIC -->|Used by| LLM_MAPPER
    CMD_BRIDGE -->|Forwards Commands| LLM_MAPPER
    LLM_MAPPER -->|Publishes| TARGET_TOPIC
    TARGET_TOPIC -->|Subscribes| MOVE_GROUP
    MOVE_GROUP -->|Plans Trajectory| SPAWN_CTRL
    SPAWN_CTRL -->|Controls| ISSIM
    MOVE_GROUP -->|Publishes| STATUS_TOPIC
    STATUS_TOPIC -->|Subscribes| TARGET_PUB

    %% Chatbot ↔ LLM Service
    CMD_BRIDGE -->|HTTP/WebSocket| Ollama["Ollama LLM"]:::external

    %% Click Events
    click USD_SCENE "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/SPARC.usd"
    click CAMERA_XACRO "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/robot_description/urdf/camera/camera.xacro"
    click YOLO_PUB "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/yolov8obb_object_detection/yolov8obb_object_detection/yolov8_obb_publisher.py"
    click YOLO_SUB "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/yolov8obb_object_detection/yolov8obb_object_detection/yolov8_obb_subscriber.py"
    click YOLO_LAUNCH "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/yolov8obb_object_detection/launch/yolov8_obb_launch.py"
    click INFERENCE_RESULT "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/yolov8_obb_msgs/msg/InferenceResult.msg"
    click YOLO_INFERENCE_MSG "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/yolov8_obb_msgs/msg/Yolov8Inference.msg"
    click PICKPLACE_STATUS "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/yolov8_obb_msgs/msg/PickPlaceStatus.msg"
    click TARGET_PUB "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/pick_place_chatbot_ui/target_publisher.py"
    click CMD_BRIDGE "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/pick_place_chatbot_ui/cmd_bridge.py"
    click LLM_MAPPER "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/pick_place_chatbot_ui/llm_mapper.py"
    click INDEX_HTML "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/pick_place_chatbot_ui/index.html"
    click LAUNCH_UI "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/pick_place_chatbot_ui/launch.sh"
    click MOVE_GROUP "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/ur5_moveit_config/launch/move_group.launch.py"
    click DEMO_LAUNCH "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/ur5_moveit_config/launch/demo.launch.py"
    click SPAWN_CTRL "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/ur5_moveit_config/launch/spawn_controllers.launch.py"
    click UR5_XACRO "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/ur5_moveit_config/config/ur5.urdf.xacro"
    click SRDF "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/ur5_moveit_config/config/ur5.srdf"
    click ROS2_CONTROL_CFG "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/ur5_moveit_config/config/moveit_controllers.yaml"
    click CPP_PICK_R "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/ur5_moveit_config/src/ur5_pick_place_cpp_r.cpp"
    click CPP_PICK_S "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/ur5_moveit_config/src/ur5_pick_place_cpp_s.cpp"
    click UR5_URDF "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/robot_description/urdf/ur5.urdf.xacro"
    click GRIPPER_XACRO "https://github.com/sahilrajpurkar03/nlp-pnp-robotic-arm/blob/master/src/robot_description/urdf/robotiq/robotiq_85_gripper.xacro"

    %% Styles
    classDef sim fill:#D0E8FF,stroke:#1E70BF,color:#1E70BF;
    classDef perception fill:#FFD6D6,stroke:#BF1E1E,color:#BF1E1E;
    classDef planning fill:#FFE4B3,stroke:#BF7F1E,color:#BF7F1E;
    classDef ui fill:#D6FFD6,stroke:#1E7F1E,color:#1E7F1E;
    classDef msgs fill:#D6F0FF,stroke:#1E7FBF,color:#1E7FBF;
    classDef core fill:#F2F2F2,stroke:#999999,color:#666666;
    classDef external fill:#EDEDED,stroke:#AAAAAA,color:#555555;
    classDef topic fill:#FFFFFF,stroke:#333333,stroke-dasharray: 5 5;
    classDef desc fill:#F0E6FF,stroke:#7F1EBF,color:#7F1EBF;
    classDef rec fill:#F5E6FF,stroke:#931EBF,color:#931EBF;
```
