How www-ye/ego2robot_blog works
Ego2Robot synthesizes robot-training demonstrations from annotated egocentric datasets or raw video. The principal workflow selects a data-source adapter, runs the documented processing stages, and renders retargeted robot trajectories; outputs can then be exported to LeRobot format. RoboTwin is a separate evaluation application with task environments, simulated robots, camera observations, and end-effector control. The README documents stage order and capabilities; some processor and evaluation internals were not sampled, so their detailed wiring is omitted.
Main entry points: pipeline.py Pipeline Driver · hand_processor.py Hand Reconstruction · depth_pose_processor.py Depth And Pose · _base_task.py Task Environments
Components 28
Synthesis Pipeline
- Pipeline Driver
ego2robot/pipeline.py - Base Processor
ego2robot/processors/base_processor.py - EgoDex Adapter
ego2robot/processors/data_sources/egodex_processor.py - EgoVerse Adapter
ego2robot/processors/data_sources/egoverse_processor.py - ViTRA Adapter
ego2robot/processors/data_sources/vitra_processor.py - Action Retargeting
ego2robot/processors/action_processor.py - Trajectory Smoothing
ego2robot/processors/smoothing_processor.py - LeRobot Export
scripts/convert_to_lerobot.py
Video Perception
- Subtask Detection
ego2robot/processors/subtask_processor.py - Depth And Pose
ego2robot/processors/depth_pose_processor.py - Hand Reconstruction
ego2robot/processors/hand_processor.py - AnyHand Model
ego2robot/models/anyhand_wrapper.py - WiLoR Model
ego2robot/models/wilor_wrapper.py - Depth Anything
ego2robot/models/da3_wrapper.py - Arm Segmentation
ego2robot/processors/segmentation_processor.py - Hand Inpainting
ego2robot/processors/handinpaint_processor.py
Robot Retargeting
- Robot Rendering
ego2robot/processors/robot_inpaint/processor.py - MuJoCo Twin
ego2robot/processors/robot_inpaint/mujoco_twin_robot.py - Camera Calibration
ego2robot/processors/calibration_ctrl.py
RoboTwin Evaluation
- Task Environments
evaluation/robotwin/envs/_base_task.py - Simulation Cameras
evaluation/robotwin/envs/camera/camera.py - Robot Planning
evaluation/robotwin/envs/robot/robot.py - Motion Planner
evaluation/robotwin/envs/robot/planner.py - EEF Controller
evaluation/robotwin/eef_control/controller.py - IK And FK Solver
evaluation/robotwin/eef_control/mink_ik_solver.py - Policy Evaluation
evaluation/robotwin/script/eval_policy.py
Other components
- Dataset User
- Policy Evaluator
Connections 26
- Dataset User → Pipeline Driver: runs
README.md - Pipeline Driver → Base Processor: orchestrates
ego2robot/pipeline.py - Pipeline Driver → EgoDex Adapter: selects source
README.md - Pipeline Driver → EgoVerse Adapter: selects source
README.md - Pipeline Driver → ViTRA Adapter: selects source
README.md - Pipeline Driver → Subtask Detection: runs first
README.md - Subtask Detection → Depth And Pose: feeds
README.md - Depth And Pose → Hand Reconstruction: feeds
README.md - Hand Reconstruction → Action Retargeting: feeds
README.md - Pipeline Driver → Action Retargeting: runs stage
README.md - Action Retargeting → Trajectory Smoothing: feeds
README.md - Trajectory Smoothing → Arm Segmentation: feeds
README.md - Arm Segmentation → Hand Inpainting: feeds
README.md - Hand Inpainting → Robot Rendering: feeds
README.md - Hand Reconstruction → AnyHand Model: uses
ego2robot/processors/hand_processor.py - Hand Reconstruction → WiLoR Model: uses
ego2robot/processors/hand_processor.py - Depth And Pose → Depth Anything: uses
- Robot Rendering → MuJoCo Twin: delegates to
ego2robot/processors/robot_inpaint/processor.py - MuJoCo Twin → Camera Calibration: uses
ego2robot/processors/robot_inpaint/mujoco_twin_robot.py - Dataset User → LeRobot Export: exports data
README.md - Policy Evaluator → Policy Evaluation: runs
- Policy Evaluation → Task Environments: evaluates in
- Task Environments → Simulation Cameras: uses
evaluation/robotwin/envs/_base_task.py - Task Environments → Robot Planning: uses
evaluation/robotwin/envs/_base_task.py - Robot Planning → Motion Planner: dispatches to
evaluation/robotwin/envs/robot/robot.py - EEF Controller → IK And FK Solver: uses
evaluation/robotwin/eef_control/controller.py