How isaac-sim/sim-to-real-so-101-workshop works
This workshop codebase supports an SO-101 sim-to-real workflow: configure Isaac environments for teleoperation or vial-to-rack tasks, collect LeRobot demonstrations, evaluate policies, and connect simulation or robot control to a remote GR00T inference server. Task configuration combines robot scenes, camera observations, reset randomization, and task-specific grasp/placement signals. README capabilities also include debug agents, domain-randomized evaluation, and dataset publishing. Some script and integration wiring is inferred only from documented capabilities and paths; sampled excerpts do not show every runtime call.
Main entry points: tasks Isaac Environments · lerobot_agent.py LeRobot Teleop · server_client.py GR00T Client · so101_env_cfg.py SO-101 Scene
Components 21
Simulation Tasks
- Isaac Environments
source/sim_to_real_so101/tasks - SO-101 Scene
source/sim_to_real_so101/tasks/so101_env_cfg.py - Vial Rack Task
source/sim_to_real_so101/tasks/vials_to_rack_env_cfg.py - Task Scene & Cameras
source/sim_to_real_so101/tasks/task_env_cfg.py - MDP Terms
source/sim_to_real_so101/mdp - Robot & Scene Assets
source/sim_to_real_so101/assets
Teleop & Data
- LeRobot Teleop
source/sim_to_real_so101/scripts/lerobot_agent.py - Keyboard Controls
source/sim_to_real_so101/utils/keyboard.py - LeRobot Interface
source/sim_to_real_so101/utils/lerobot_interface.py - Episode Recorder
source/sim_to_real_so101/utils/lerobot_recorder.py - LeRobot Dataset
- Dataset Publisher
source/sim_to_real_so101/scripts/lerobot_push_dataset.py
Policy & Evaluation
- Policy Contracts
source/sim_to_real_so101/gr00t_client/policy.py - GR00T Client
source/sim_to_real_so101/gr00t_client/server_client.py - Policy Evaluation
source/sim_to_real_so101/scripts/lerobot_eval.py - Debug Agents
source/sim_to_real_so101/scripts
Robot Deployment
- Real Robot Tools
docker/real/scripts - GR00T Inference Server
Other components
- Workshop User
- Robot Operator
- Dataset Hub
Connections 27
- Workshop User → LeRobot Teleop: starts teleop
- Workshop User → Policy Evaluation: runs evaluation
- Workshop User → Debug Agents: runs debug
- SO-101 Scene → Isaac Environments: configures robot
- Task Scene & Cameras → Isaac Environments: configures scene
- Vial Rack Task → Isaac Environments: configures task
- Vial Rack Task → MDP Terms: uses task signals
- SO-101 Scene → MDP Terms: uses MDP terms
- SO-101 Scene → Robot & Scene Assets: loads assets
- Task Scene & Cameras → Robot & Scene Assets: loads scene assets
- LeRobot Teleop → Isaac Environments: controls simulation
- LeRobot Teleop → LeRobot Interface: uses interface
- LeRobot Teleop → Keyboard Controls: handles controls
- LeRobot Teleop → Episode Recorder: records episodes
- Keyboard Controls → Episode Recorder: dispatches events
- Episode Recorder → LeRobot Dataset: writes episodes
- Policy Evaluation → Isaac Environments: evaluates in
- Policy Evaluation → Policy Contracts: uses policy API
- Debug Agents → Isaac Environments: supplies actions
- Dataset Publisher → LeRobot Dataset: reads dataset
- Dataset Publisher → Dataset Hub: publishes dataset
- Robot Operator → Real Robot Tools: runs robot tools
- Real Robot Tools → LeRobot Interface: uses robot interface
- LeRobot Interface → GR00T Client: requests actions
- GR00T Client → GR00T Inference Server: sends observations
- GR00T Inference Server → GR00T Client: returns actions
- GR00T Client → Policy Contracts: implements policy