How crane-1/ros2-isaacsim-lerobot-dataset-collection works
This repository supports a dual-arm object-transfer workflow: Isaac Sim scenes and robot assets provide the simulation context; ROS 2 scripts collect camera, joint-state, and action episodes into LeRobot-style datasets; dataset utilities convert and summarize those datasets; LeRobot fine-tuning and ACT/SmolVLA ROS inference complete the training-and-evaluation loop. Collection and inference details are based on sampled scripts; training is described by the README but its implementation was not sampled.
Main entry points: scripts Episode Dataset · lerobot_related LeRobot Fine-tuning · data_collection.py Single-Arm Collector · eval_act.py ACT ROS Inference
Components 18
Simulation Assets
- Robot Scenes
Grid - Piper Models
USD
ROS Data Collection
- Single-Arm Collector
src/data_collection/scripts/data_collection.py - Dual-Arm Collector
src/data_collection/scripts/dual_collection.py
Dataset Processing
- Episode Dataset
src/data_collection/scripts - Episode Metadata
lerobot_related/create_episodes_jsonl.py - Dataset Statistics
lerobot_related/create_episodes_stats_jsonl.py - Dataset Converter
lerobot_related/convert.py
Training and Inference
- LeRobot Fine-tuning
lerobot_related - ACT ROS Inference
lerobot_related/eval_act.py - Dual ACT Inference
lerobot_related/eval_act_dual.py - SmolVLA ROS Inference
lerobot_related/eval_smolvla.py - Dual SmolVLA Inference
lerobot_related/smol_vla_dual.py
Evaluation Results
- ACT Evaluation Artifacts
pics/act_result - SmolVLA Evaluation Artifacts
pics/smol_vla_result
Other components
- Task Operator
- Isaac Sim
- Robot / Simulator Control
Connections 30
- Task Operator → Single-Arm Collector: starts recording
- Task Operator → Dual-Arm Collector: starts recording
- Isaac Sim → Robot Scenes: loads assets
- Isaac Sim → Piper Models: loads assets
- Isaac Sim → Single-Arm Collector: publishes ROS data
- Isaac Sim → Dual-Arm Collector: publishes ROS data
- Single-Arm Collector → Robot / Simulator Control: publishes joint states
- Single-Arm Collector → Episode Dataset: records episodes
- Dual-Arm Collector → Episode Dataset: records episodes
- Episode Dataset → Episode Metadata: reads episodes
- Episode Dataset → Dataset Statistics: reads episodes
- Episode Dataset → Dataset Converter: converts dataset
- Episode Dataset → LeRobot Fine-tuning: trains from data
- Dataset Converter → LeRobot Fine-tuning: provides dataset
- LeRobot Fine-tuning → ACT ROS Inference: provides checkpoint
- LeRobot Fine-tuning → Dual ACT Inference: provides checkpoint
- LeRobot Fine-tuning → SmolVLA ROS Inference: provides checkpoint
- LeRobot Fine-tuning → Dual SmolVLA Inference: provides checkpoint
- Isaac Sim → ACT ROS Inference: publishes observations
- Isaac Sim → Dual ACT Inference: publishes observations
- Isaac Sim → SmolVLA ROS Inference: publishes observations
- Isaac Sim → Dual SmolVLA Inference: publishes observations
- ACT ROS Inference → Robot / Simulator Control: publishes actions
- Dual ACT Inference → Robot / Simulator Control: publishes actions
- SmolVLA ROS Inference → Robot / Simulator Control: publishes actions
- Dual SmolVLA Inference → Robot / Simulator Control: publishes actions
- ACT ROS Inference → ACT Evaluation Artifacts: evaluated in
- Dual ACT Inference → ACT Evaluation Artifacts: evaluated in
- SmolVLA ROS Inference → SmolVLA Evaluation Artifacts: evaluated in
- Dual SmolVLA Inference → SmolVLA Evaluation Artifacts: evaluated in