# isaac-sim/sim-to-real-so-101-workshop architecture

> Architecture diagram of the GitHub repository isaac-sim/sim-to-real-so-101-workshop, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/isaac-sim/sim-to-real-so-101-workshop
- Repository: https://github.com/isaac-sim/sim-to-real-so-101-workshop

## Overview

This workshop codebase supports an SO-101 sim-to-real workflow: configure Isaac environments for teleoperation or vial-to-rack tasks, collect LeRobot demonstrations, evaluate policies, and connect simulation or robot control to a remote GR00T inference server. Task configuration combines robot scenes, camera observations, reset randomization, and task-specific grasp/placement signals. README capabilities also include debug agents, domain-randomized evaluation, and dataset publishing. Some script and integration wiring is inferred only from documented capabilities and paths; sampled excerpts do not show every runtime call.

Main entry points: [`source/sim_to_real_so101/tasks`](https://github.com/isaac-sim/sim-to-real-so-101-workshop/tree/main/source/sim_to_real_so101/tasks) (Isaac Environments), [`source/sim_to_real_so101/scripts/lerobot_agent.py`](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/scripts/lerobot_agent.py) (LeRobot Teleop), [`source/sim_to_real_so101/gr00t_client/server_client.py`](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/gr00t_client/server_client.py) (GR00T Client), [`source/sim_to_real_so101/tasks/so101_env_cfg.py`](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/tasks/so101_env_cfg.py) (SO-101 Scene)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_simulation["Simulation Tasks"]
  node_sim_env["Isaac Environments"]
  node_robot_cfg["SO-101 Scene<br/>[so101_env_cfg.py]"]
  node_task_cfg["Vial Rack Task"]
  node_task_base["Task Scene &amp; Cameras<br/>[task_env_cfg.py]"]
  node_mdp["MDP Terms"]
  node_assets["Robot &amp; Scene Assets"]
end

subgraph group_teleoperation["Teleop &amp; Data"]
  node_teleop_agent["LeRobot Teleop<br/>[lerobot_agent.py]"]
  node_keyboard["Keyboard Controls<br/>[keyboard.py]"]
  node_robot_iface["LeRobot Interface"]
  node_recorder["Episode Recorder"]
  node_dataset[("LeRobot Dataset")]
  node_dataset_push["Dataset Publisher"]
end

subgraph group_inference["Policy &amp; Evaluation"]
  node_policy["Policy Contracts<br/>[policy.py]"]
  node_remote_client["GR00T Client<br/>[server_client.py]"]
  node_evaluation["Policy Evaluation<br/>[lerobot_eval.py]"]
  node_debug_agents["Debug Agents"]
end

subgraph group_deployment["Robot Deployment"]
  node_real_scripts["Real Robot Tools"]
  node_inference_server["GR00T Inference Server"]
end

node_user(("Workshop User"))
node_robot_operator(("Robot Operator"))
node_dataset_hub[("Dataset Hub")]

node_user -->|"starts teleop"| node_teleop_agent
node_user -->|"runs evaluation"| node_evaluation
node_user -->|"runs debug"| node_debug_agents
node_robot_cfg -->|"configures robot"| node_sim_env
node_task_base -->|"configures scene"| node_sim_env
node_task_cfg -->|"configures task"| node_sim_env
node_task_cfg -->|"uses task signals"| node_mdp
node_robot_cfg -->|"uses MDP terms"| node_mdp
node_robot_cfg -->|"loads assets"| node_assets
node_task_base -->|"loads scene assets"| node_assets
node_teleop_agent -->|"controls simulation"| node_sim_env
node_teleop_agent -->|"uses interface"| node_robot_iface
node_teleop_agent -->|"handles controls"| node_keyboard
node_teleop_agent -->|"records episodes"| node_recorder
node_keyboard -->|"dispatches events"| node_recorder
node_recorder -->|"writes episodes"| node_dataset
node_evaluation -->|"evaluates in"| node_sim_env
node_evaluation -->|"uses policy API"| node_policy
node_debug_agents -->|"supplies actions"| node_sim_env
node_dataset_push -->|"reads dataset"| node_dataset
node_dataset_push -->|"publishes dataset"| node_dataset_hub
node_robot_operator -->|"runs robot tools"| node_real_scripts
node_real_scripts -->|"uses robot interface"| node_robot_iface
node_robot_iface -->|"requests actions"| node_remote_client
node_remote_client -->|"sends observations"| node_inference_server
node_inference_server -->|"returns actions"| node_remote_client
node_remote_client -->|"implements policy"| node_policy

click node_sim_env "https://github.com/isaac-sim/sim-to-real-so-101-workshop/tree/main/source/sim_to_real_so101/tasks"
click node_robot_cfg "https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/tasks/so101_env_cfg.py"
click node_task_cfg "https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/tasks/vials_to_rack_env_cfg.py"
click node_task_base "https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/tasks/task_env_cfg.py"
click node_mdp "https://github.com/isaac-sim/sim-to-real-so-101-workshop/tree/main/source/sim_to_real_so101/mdp"
click node_assets "https://github.com/isaac-sim/sim-to-real-so-101-workshop/tree/main/source/sim_to_real_so101/assets"
click node_teleop_agent "https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/scripts/lerobot_agent.py"
click node_keyboard "https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/utils/keyboard.py"
click node_robot_iface "https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/utils/lerobot_interface.py"
click node_recorder "https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/utils/lerobot_recorder.py"
click node_policy "https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/gr00t_client/policy.py"
click node_remote_client "https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/gr00t_client/server_client.py"
click node_evaluation "https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/scripts/lerobot_eval.py"
click node_debug_agents "https://github.com/isaac-sim/sim-to-real-so-101-workshop/tree/main/source/sim_to_real_so101/scripts"
click node_dataset_push "https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/scripts/lerobot_push_dataset.py"
click node_real_scripts "https://github.com/isaac-sim/sim-to-real-so-101-workshop/tree/main/docker/real/scripts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sim_env,node_robot_cfg,node_task_cfg,node_task_base,node_mdp,node_assets,node_user toneBlue
class node_teleop_agent,node_keyboard,node_robot_iface,node_recorder,node_dataset,node_dataset_push,node_dataset_hub toneAmber
class node_policy,node_remote_client,node_evaluation,node_debug_agents toneMint
class node_real_scripts,node_inference_server toneRose
class node_robot_operator toneIndigo
```

## Components

### Simulation Tasks

- [Isaac Environments](https://github.com/isaac-sim/sim-to-real-so-101-workshop/tree/main/source/sim_to_real_so101/tasks): `source/sim_to_real_so101/tasks`
- [SO-101 Scene](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/tasks/so101_env_cfg.py): `source/sim_to_real_so101/tasks/so101_env_cfg.py`
- [Vial Rack Task](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/tasks/vials_to_rack_env_cfg.py): `source/sim_to_real_so101/tasks/vials_to_rack_env_cfg.py`
- [Task Scene & Cameras](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/tasks/task_env_cfg.py): `source/sim_to_real_so101/tasks/task_env_cfg.py`
- [MDP Terms](https://github.com/isaac-sim/sim-to-real-so-101-workshop/tree/main/source/sim_to_real_so101/mdp): `source/sim_to_real_so101/mdp`
- [Robot & Scene Assets](https://github.com/isaac-sim/sim-to-real-so-101-workshop/tree/main/source/sim_to_real_so101/assets): `source/sim_to_real_so101/assets`

### Teleop & Data

- [LeRobot Teleop](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/scripts/lerobot_agent.py): `source/sim_to_real_so101/scripts/lerobot_agent.py`
- [Keyboard Controls](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/utils/keyboard.py): `source/sim_to_real_so101/utils/keyboard.py`
- [LeRobot Interface](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/utils/lerobot_interface.py): `source/sim_to_real_so101/utils/lerobot_interface.py`
- [Episode Recorder](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/utils/lerobot_recorder.py): `source/sim_to_real_so101/utils/lerobot_recorder.py`
- **LeRobot Dataset**
- [Dataset Publisher](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/scripts/lerobot_push_dataset.py): `source/sim_to_real_so101/scripts/lerobot_push_dataset.py`

### Policy & Evaluation

- [Policy Contracts](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/gr00t_client/policy.py): `source/sim_to_real_so101/gr00t_client/policy.py`
- [GR00T Client](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/gr00t_client/server_client.py): `source/sim_to_real_so101/gr00t_client/server_client.py`
- [Policy Evaluation](https://github.com/isaac-sim/sim-to-real-so-101-workshop/blob/main/source/sim_to_real_so101/scripts/lerobot_eval.py): `source/sim_to_real_so101/scripts/lerobot_eval.py`
- [Debug Agents](https://github.com/isaac-sim/sim-to-real-so-101-workshop/tree/main/source/sim_to_real_so101/scripts): `source/sim_to_real_so101/scripts`

### Robot Deployment

- [Real Robot Tools](https://github.com/isaac-sim/sim-to-real-so-101-workshop/tree/main/docker/real/scripts): `docker/real/scripts`
- **GR00T Inference Server**

### Other components

- **Workshop User**
- **Robot Operator**
- **Dataset Hub**

## Connections

- Workshop User → LeRobot Teleop: starts teleop
- Workshop User → Policy Evaluation: runs evaluation
- Workshop User → Debug Agents: runs debug
- SO-101 Scene → Isaac Environments: configures robot
- Task Scene & Cameras → Isaac Environments: configures scene
- Vial Rack Task → Isaac Environments: configures task
- Vial Rack Task → MDP Terms: uses task signals
- SO-101 Scene → MDP Terms: uses MDP terms
- SO-101 Scene → Robot & Scene Assets: loads assets
- Task Scene & Cameras → Robot & Scene Assets: loads scene assets
- LeRobot Teleop → Isaac Environments: controls simulation
- LeRobot Teleop → LeRobot Interface: uses interface
- LeRobot Teleop → Keyboard Controls: handles controls
- LeRobot Teleop → Episode Recorder: records episodes
- Keyboard Controls → Episode Recorder: dispatches events
- Episode Recorder → LeRobot Dataset: writes episodes
- Policy Evaluation → Isaac Environments: evaluates in
- Policy Evaluation → Policy Contracts: uses policy API
- Debug Agents → Isaac Environments: supplies actions
- Dataset Publisher → LeRobot Dataset: reads dataset
- Dataset Publisher → Dataset Hub: publishes dataset
- Robot Operator → Real Robot Tools: runs robot tools
- Real Robot Tools → LeRobot Interface: uses robot interface
- LeRobot Interface → GR00T Client: requests actions
- GR00T Client → GR00T Inference Server: sends observations
- GR00T Inference Server → GR00T Client: returns actions
- GR00T Client → Policy Contracts: implements policy