How edwin-metcalf/isaacsim-robot-project works
This repository evaluates language-guided pick-and-place in headless Isaac Sim. A user supplies an instruction; the evaluation workflow creates randomized cube scenes, provides scene state to the Anthropic planner, executes the resulting pick/place through a six-phase robot task, and records trial metrics. The controller is the internal motion layer beneath task execution. The README documents the overall workflow and automated JSON logging; the orchestration implementation in test_vlm.py is not excerpted, so its connections are based on that documentation.
Main entry points: test_vlm.py Trial Runner · vlm_planner.py VLM Planner · env.py Scene & State · task.py Pick-and-Place Task
Components 10
Evaluation Workflow
- Evaluation CLI
eval.py - Trial Runner
test_vlm.py
VLM Planning
- VLM Planner
vlm_planner.py
Simulation & Control
- Scene & State
env.py - Pick-and-Place Task
task.py - RMPFlow Controller
controller.py
Results
- Trial Metrics & JSON
test_vlm.py
Other components
- Instruction User
- Anthropic API
- Isaac Sim
Connections 11
- Instruction User → Evaluation CLI: provides prompt
README.md - Evaluation CLI → Trial Runner: starts evaluation
eval.py - Trial Runner → Scene & State: sets up scenes
README.md - Scene & State → VLM Planner: provides scene state
README.md - VLM Planner → Anthropic API: requests plan
vlm_planner.py - VLM Planner → Pick-and-Place Task: provides plan
README.md - Pick-and-Place Task → RMPFlow Controller: commands motion
task.py - Trial Runner → Pick-and-Place Task: executes trials
README.md - Trial Runner → Trial Metrics & JSON: records results
README.md - Scene & State → Isaac Sim: builds simulation
env.py - RMPFlow Controller → Isaac Sim: drives robot
controller.py