# edwin-metcalf/isaacsim-robot-project architecture

> Architecture diagram of the GitHub repository edwin-metcalf/isaacsim-robot-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/edwin-metcalf/isaacsim-robot-project
- Repository: https://github.com/edwin-metcalf/isaacsim-robot-project

## Overview

This repository evaluates language-guided pick-and-place in headless Isaac Sim. A user supplies an instruction; the evaluation workflow creates randomized cube scenes, provides scene state to the Anthropic planner, executes the resulting pick/place through a six-phase robot task, and records trial metrics. The controller is the internal motion layer beneath task execution. The README documents the overall workflow and automated JSON logging; the orchestration implementation in test_vlm.py is not excerpted, so its connections are based on that documentation.

Main entry points: [`test_vlm.py`](https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/test_vlm.py) (Trial Runner), [`vlm_planner.py`](https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/vlm_planner.py) (VLM Planner), [`env.py`](https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/env.py) (Scene & State), [`task.py`](https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/task.py) (Pick-and-Place Task)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Evaluation Workflow"]
  node_cli["Evaluation CLI<br/>[eval.py]"]
  node_runner["Trial Runner<br/>[test_vlm.py]"]
end

subgraph group_planning["VLM Planning"]
  node_planner["VLM Planner<br/>[vlm_planner.py]"]
end

subgraph group_simulation["Simulation &amp; Control"]
  node_scene["Scene &amp; State<br/>[env.py]"]
  node_task["Pick-and-Place Task<br/>[task.py]"]
  node_controller["RMPFlow Controller<br/>[controller.py]"]
end

subgraph group_evaluation["Results"]
  node_metrics["Trial Metrics &amp; JSON<br/>[test_vlm.py]"]
end

node_user(("Instruction User"))
node_anthropic{{"Anthropic API"}}
node_isaac{{"Isaac Sim"}}

node_user -->|"provides prompt"| node_cli
node_cli -->|"starts evaluation"| node_runner
node_runner -->|"sets up scenes"| node_scene
node_scene -->|"provides scene state"| node_planner
node_planner -->|"requests plan"| node_anthropic
node_planner -->|"provides plan"| node_task
node_task -->|"commands motion"| node_controller
node_runner -->|"executes trials"| node_task
node_runner -->|"records results"| node_metrics
node_scene -->|"builds simulation"| node_isaac
node_controller -->|"drives robot"| node_isaac

click node_cli "https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/eval.py"
click node_runner "https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/test_vlm.py"
click node_scene "https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/env.py"
click node_planner "https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/vlm_planner.py"
click node_task "https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/task.py"
click node_controller "https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/controller.py"
click node_metrics "https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/test_vlm.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_runner,node_user toneBlue
class node_planner toneAmber
class node_scene,node_task,node_controller,node_anthropic toneMint
class node_metrics toneRose
class node_isaac toneIndigo
```

## Components

### Evaluation Workflow

- [Evaluation CLI](https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/eval.py): `eval.py`
- [Trial Runner](https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/test_vlm.py): `test_vlm.py`

### VLM Planning

- [VLM Planner](https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/vlm_planner.py): `vlm_planner.py`

### Simulation & Control

- [Scene & State](https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/env.py): `env.py`
- [Pick-and-Place Task](https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/task.py): `task.py`
- [RMPFlow Controller](https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/controller.py): `controller.py`

### Results

- [Trial Metrics & JSON](https://github.com/edwin-metcalf/isaacsim-robot-project/blob/main/test_vlm.py): `test_vlm.py`

### Other components

- **Instruction User**
- **Anthropic API**
- **Isaac Sim**

## Connections

- Instruction User → Evaluation CLI: provides prompt (evidence: `README.md`)
- Evaluation CLI → Trial Runner: starts evaluation (evidence: `eval.py`)
- Trial Runner → Scene & State: sets up scenes (evidence: `README.md`)
- Scene & State → VLM Planner: provides scene state (evidence: `README.md`)
- VLM Planner → Anthropic API: requests plan (evidence: `vlm_planner.py`)
- VLM Planner → Pick-and-Place Task: provides plan (evidence: `README.md`)
- Pick-and-Place Task → RMPFlow Controller: commands motion (evidence: `task.py`)
- Trial Runner → Pick-and-Place Task: executes trials (evidence: `README.md`)
- Trial Runner → Trial Metrics & JSON: records results (evidence: `README.md`)
- Scene & State → Isaac Sim: builds simulation (evidence: `env.py`)
- RMPFlow Controller → Isaac Sim: drives robot (evidence: `controller.py`)
