How roopshub/rlbench-multi-agent works
This repository is a human-approved, language-guided robotic manipulation framework for RLBench. A user submits a task; the root agent plans adjustable steps and waits for approval, then coordinates sensing, object localization, and robot motion. The RLBench MCP server provides camera and manipulation tools, while the perception server detects prompted objects and derives 3D positions. The README also documents supported task types and parameter editing. The batch runner exercises the agent and records evaluation results. Edges reflect README workflow statements and shown server configuration; implementation detail beyond the excerpts is not inferred.
Main entry points: agent.py Root orchestrator · rlbench_orchestration_server.py Camera sensing · perception_orchestration_server.py 3D object detection · batch_runner.py Batch evaluation runner
Components 11
Planning and approval
- Root orchestrator
multi_tool_agent/agent.py - Plan and parameters
multi_tool_agent/agent.py - Human approval
multi_tool_agent/agent.py
Sensing and motion
- Camera sensing
multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py - Cartesian and gripper control
multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py
Object perception
- 3D object detection
multi_tool_agent/ros_mcp_server/perception_orchestration_server.py
Evaluation
- Batch evaluation runner
multi_tool_agent/evaluation/batch_runner.py - Evaluation results
multi_tool_agent/evaluation/results
Other components
- Task requester
- RLBench simulator
- GroundingDINO
Connections 13
- Task requester → Root orchestrator: submits task
README.md - Root orchestrator → Plan and parameters: generates plan
README.md - Plan and parameters → Human approval: requests approval
README.md - Human approval → Root orchestrator: approves plan
README.md - Root orchestrator → Camera sensing: requests observations
multi_tool_agent/agent.py - Camera sensing → 3D object detection: provides sensor data
README.md - Root orchestrator → 3D object detection: dispatches detection
multi_tool_agent/agent.py - Root orchestrator → Cartesian and gripper control: executes approved steps
README.md - Camera sensing → RLBench simulator: captures observations
multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py - Cartesian and gripper control → RLBench simulator: controls robot
multi_tool_agent/ros_mcp_server/rlbench_orchestration_server.py - 3D object detection → GroundingDINO: detects prompted objects
README.md - Batch evaluation runner → Root orchestrator: runs trials
multi_tool_agent/evaluation/batch_runner.py - Batch evaluation runner → Evaluation results: writes metrics
multi_tool_agent/evaluation/batch_runner.py