# roopshub/rlbench-multi-agent architecture

> Architecture diagram of the GitHub repository roopshub/rlbench-multi-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/roopshub/rlbench-multi-agent
- Repository: https://github.com/roopshub/rlbench-multi-agent

## Overview

This repository is a human-approved, language-guided robotic manipulation framework for RLBench. A user submits a task; the root agent plans adjustable steps and waits for approval, then coordinates sensing, object localization, and robot motion. The RLBench MCP server provides camera and manipulation tools, while the perception server detects prompted objects and derives 3D positions. The README also documents supported task types and parameter editing. The batch runner exercises the agent and records evaluation results. Edges reflect README workflow statements and shown server configuration; implementation detail beyond the excerpts is not inferred.

Main entry points: [`multi_tool_agent/agent.py`](https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/agent.py) (Root orchestrator), [`multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py`](https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py) (Camera sensing), [`multi_tool_agent/ros_mcp_server/perception_orchestration_server.py`](https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/ros_mcp_server/perception_orchestration_server.py) (3D object detection), [`multi_tool_agent/evaluation/batch_runner.py`](https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/evaluation/batch_runner.py) (Batch evaluation runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_coordination["Planning and approval"]
  node_root["Root orchestrator<br/>[agent.py]"]
  node_planner["Plan and parameters<br/>[agent.py]"]
  node_approval["Human approval<br/>[agent.py]"]
end

subgraph group_robot["Sensing and motion"]
  node_sensing["Camera sensing"]
  node_motion["Cartesian and gripper control"]
end

subgraph group_vision["Object perception"]
  node_perception["3D object detection"]
end

subgraph group_evaluation["Evaluation"]
  node_batch["Batch evaluation runner<br/>[batch_runner.py]"]
  node_results["Evaluation results"]
end

node_user(("Task requester"))
node_simulator["RLBench simulator"]
node_detector["GroundingDINO"]

node_user -->|"submits task"| node_root
node_root -->|"generates plan"| node_planner
node_planner -->|"requests approval"| node_approval
node_approval -->|"approves plan"| node_root
node_root -->|"requests observations"| node_sensing
node_sensing -->|"provides sensor data"| node_perception
node_root -->|"dispatches detection"| node_perception
node_root -->|"executes approved steps"| node_motion
node_sensing -->|"captures observations"| node_simulator
node_motion -->|"controls robot"| node_simulator
node_perception -->|"detects prompted objects"| node_detector
node_batch -->|"runs trials"| node_root
node_batch -->|"writes metrics"| node_results

click node_root "https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/agent.py"
click node_planner "https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/agent.py"
click node_approval "https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/agent.py"
click node_sensing "https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py"
click node_perception "https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/ros_mcp_server/perception_orchestration_server.py"
click node_motion "https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py"
click node_batch "https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/evaluation/batch_runner.py"
click node_results "https://github.com/roopshub/rlbench-multi-agent/tree/main/multi_tool_agent/evaluation/results"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_root,node_planner,node_approval toneBlue
class node_sensing,node_motion toneAmber
class node_perception toneMint
class node_batch,node_results,node_user toneRose
class node_simulator,node_detector toneIndigo
```

## Components

### Planning and approval

- [Root orchestrator](https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/agent.py): `multi_tool_agent/agent.py`
- [Plan and parameters](https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/agent.py): `multi_tool_agent/agent.py`
- [Human approval](https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/agent.py): `multi_tool_agent/agent.py`

### Sensing and motion

- [Camera sensing](https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py): `multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py`
- [Cartesian and gripper control](https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py): `multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py`

### Object perception

- [3D object detection](https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/ros_mcp_server/perception_orchestration_server.py): `multi_tool_agent/ros_mcp_server/perception_orchestration_server.py`

### Evaluation

- [Batch evaluation runner](https://github.com/roopshub/rlbench-multi-agent/blob/main/multi_tool_agent/evaluation/batch_runner.py): `multi_tool_agent/evaluation/batch_runner.py`
- [Evaluation results](https://github.com/roopshub/rlbench-multi-agent/tree/main/multi_tool_agent/evaluation/results): `multi_tool_agent/evaluation/results`

### Other components

- **Task requester**
- **RLBench simulator**
- **GroundingDINO**

## Connections

- Task requester → Root orchestrator: submits task (evidence: `README.md`)
- Root orchestrator → Plan and parameters: generates plan (evidence: `README.md`)
- Plan and parameters → Human approval: requests approval (evidence: `README.md`)
- Human approval → Root orchestrator: approves plan (evidence: `README.md`)
- Root orchestrator → Camera sensing: requests observations (evidence: `multi_tool_agent/agent.py`)
- Camera sensing → 3D object detection: provides sensor data (evidence: `README.md`)
- Root orchestrator → 3D object detection: dispatches detection (evidence: `multi_tool_agent/agent.py`)
- Root orchestrator → Cartesian and gripper control: executes approved steps (evidence: `README.md`)
- Camera sensing → RLBench simulator: captures observations (evidence: `multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py`)
- Cartesian and gripper control → RLBench simulator: controls robot (evidence: `multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py`)
- 3D object detection → GroundingDINO: detects prompted objects (evidence: `README.md`)
- Batch evaluation runner → Root orchestrator: runs trials (evidence: `multi_tool_agent/evaluation/batch_runner.py`)
- Batch evaluation runner → Evaluation results: writes metrics (evidence: `multi_tool_agent/evaluation/batch_runner.py`)
