# www-ye/ego2robot_blog architecture

> Architecture diagram of the GitHub repository www-ye/ego2robot_blog, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/www-ye/ego2robot_blog
- Repository: https://github.com/www-ye/ego2robot_blog

## Overview

Ego2Robot synthesizes robot-training demonstrations from annotated egocentric datasets or raw video. The principal workflow selects a data-source adapter, runs the documented processing stages, and renders retargeted robot trajectories; outputs can then be exported to LeRobot format. RoboTwin is a separate evaluation application with task environments, simulated robots, camera observations, and end-effector control. The README documents stage order and capabilities; some processor and evaluation internals were not sampled, so their detailed wiring is omitted.

Main entry points: [`ego2robot/pipeline.py`](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/pipeline.py) (Pipeline Driver), [`ego2robot/processors/hand_processor.py`](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/hand_processor.py) (Hand Reconstruction), [`ego2robot/processors/depth_pose_processor.py`](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/depth_pose_processor.py) (Depth And Pose), [`evaluation/robotwin/envs/_base_task.py`](https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/envs/_base_task.py) (Task Environments)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_synthesis["Synthesis Pipeline"]
  node_driver["Pipeline Driver<br/>[pipeline.py]"]
  node_base["Base Processor<br/>[base_processor.py]"]
  node_egodex["EgoDex Adapter"]
  node_egoverse["EgoVerse Adapter"]
  node_vitra["ViTRA Adapter<br/>[vitra_processor.py]"]
  node_action["Action Retargeting"]
  node_smoothing["Trajectory Smoothing"]
  node_export["LeRobot Export"]
end

subgraph group_perception["Video Perception"]
  node_subtask["Subtask Detection"]
  node_depth["Depth And Pose"]
  node_hand["Hand Reconstruction<br/>[hand_processor.py]"]
  node_anyhand["AnyHand Model<br/>[anyhand_wrapper.py]"]
  node_wilor["WiLoR Model<br/>[wilor_wrapper.py]"]
  node_da3["Depth Anything<br/>[da3_wrapper.py]"]
  node_segmentation["Arm Segmentation"]
  node_handinpaint["Hand Inpainting"]
end

subgraph group_render["Robot Retargeting"]
  node_robotinpaint["Robot Rendering<br/>[processor.py]"]
  node_twin["MuJoCo Twin"]
  node_calibration["Camera Calibration"]
end

subgraph group_evaluation["RoboTwin Evaluation"]
  node_tasks["Task Environments<br/>[_base_task.py]"]
  node_camera["Simulation Cameras<br/>[camera.py]"]
  node_robot["Robot Planning<br/>[robot.py]"]
  node_planner["Motion Planner<br/>[planner.py]"]
  node_eef["EEF Controller<br/>[controller.py]"]
  node_ik["IK And FK Solver<br/>[mink_ik_solver.py]"]
  node_evalscript["Policy Evaluation<br/>[eval_policy.py]"]
end

node_user(("Dataset User"))
node_evaluser(("Policy Evaluator"))

node_user -->|"runs"| node_driver
node_driver -->|"orchestrates"| node_base
node_driver -->|"selects source"| node_egodex
node_driver -->|"selects source"| node_egoverse
node_driver -->|"selects source"| node_vitra
node_driver -->|"runs first"| node_subtask
node_subtask -->|"feeds"| node_depth
node_depth -->|"feeds"| node_hand
node_hand -->|"feeds"| node_action
node_driver -->|"runs stage"| node_action
node_action -->|"feeds"| node_smoothing
node_smoothing -->|"feeds"| node_segmentation
node_segmentation -->|"feeds"| node_handinpaint
node_handinpaint -->|"feeds"| node_robotinpaint
node_hand -->|"uses"| node_anyhand
node_hand -->|"uses"| node_wilor
node_depth -.->|"uses"| node_da3
node_robotinpaint -->|"delegates to"| node_twin
node_twin -->|"uses"| node_calibration
node_user -->|"exports data"| node_export
node_evaluser -.->|"runs"| node_evalscript
node_evalscript -.->|"evaluates in"| node_tasks
node_tasks -->|"uses"| node_camera
node_tasks -->|"uses"| node_robot
node_robot -->|"dispatches to"| node_planner
node_eef -->|"uses"| node_ik

click node_driver "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/pipeline.py"
click node_base "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/base_processor.py"
click node_egodex "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/data_sources/egodex_processor.py"
click node_egoverse "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/data_sources/egoverse_processor.py"
click node_vitra "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/data_sources/vitra_processor.py"
click node_subtask "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/subtask_processor.py"
click node_depth "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/depth_pose_processor.py"
click node_hand "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/hand_processor.py"
click node_anyhand "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/models/anyhand_wrapper.py"
click node_wilor "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/models/wilor_wrapper.py"
click node_da3 "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/models/da3_wrapper.py"
click node_action "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/action_processor.py"
click node_smoothing "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/smoothing_processor.py"
click node_segmentation "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/segmentation_processor.py"
click node_handinpaint "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/handinpaint_processor.py"
click node_robotinpaint "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/robot_inpaint/processor.py"
click node_twin "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/robot_inpaint/mujoco_twin_robot.py"
click node_calibration "https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/calibration_ctrl.py"
click node_export "https://github.com/www-ye/ego2robot_blog/blob/main/scripts/convert_to_lerobot.py"
click node_tasks "https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/envs/_base_task.py"
click node_camera "https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/envs/camera/camera.py"
click node_robot "https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/envs/robot/robot.py"
click node_planner "https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/envs/robot/planner.py"
click node_eef "https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/eef_control/controller.py"
click node_ik "https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/eef_control/mink_ik_solver.py"
click node_evalscript "https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/script/eval_policy.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_driver,node_base,node_egodex,node_egoverse,node_vitra,node_action,node_smoothing,node_export,node_user toneBlue
class node_subtask,node_depth,node_hand,node_anyhand,node_wilor,node_da3,node_segmentation,node_handinpaint toneAmber
class node_robotinpaint,node_twin,node_calibration toneMint
class node_tasks,node_camera,node_robot,node_planner,node_eef,node_ik,node_evalscript toneRose
class node_evaluser toneIndigo
```

## Components

### Synthesis Pipeline

- [Pipeline Driver](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/pipeline.py): `ego2robot/pipeline.py`
- [Base Processor](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/base_processor.py): `ego2robot/processors/base_processor.py`
- [EgoDex Adapter](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/data_sources/egodex_processor.py): `ego2robot/processors/data_sources/egodex_processor.py`
- [EgoVerse Adapter](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/data_sources/egoverse_processor.py): `ego2robot/processors/data_sources/egoverse_processor.py`
- [ViTRA Adapter](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/data_sources/vitra_processor.py): `ego2robot/processors/data_sources/vitra_processor.py`
- [Action Retargeting](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/action_processor.py): `ego2robot/processors/action_processor.py`
- [Trajectory Smoothing](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/smoothing_processor.py): `ego2robot/processors/smoothing_processor.py`
- [LeRobot Export](https://github.com/www-ye/ego2robot_blog/blob/main/scripts/convert_to_lerobot.py): `scripts/convert_to_lerobot.py`

### Video Perception

- [Subtask Detection](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/subtask_processor.py): `ego2robot/processors/subtask_processor.py`
- [Depth And Pose](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/depth_pose_processor.py): `ego2robot/processors/depth_pose_processor.py`
- [Hand Reconstruction](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/hand_processor.py): `ego2robot/processors/hand_processor.py`
- [AnyHand Model](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/models/anyhand_wrapper.py): `ego2robot/models/anyhand_wrapper.py`
- [WiLoR Model](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/models/wilor_wrapper.py): `ego2robot/models/wilor_wrapper.py`
- [Depth Anything](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/models/da3_wrapper.py): `ego2robot/models/da3_wrapper.py`
- [Arm Segmentation](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/segmentation_processor.py): `ego2robot/processors/segmentation_processor.py`
- [Hand Inpainting](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/handinpaint_processor.py): `ego2robot/processors/handinpaint_processor.py`

### Robot Retargeting

- [Robot Rendering](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/robot_inpaint/processor.py): `ego2robot/processors/robot_inpaint/processor.py`
- [MuJoCo Twin](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/robot_inpaint/mujoco_twin_robot.py): `ego2robot/processors/robot_inpaint/mujoco_twin_robot.py`
- [Camera Calibration](https://github.com/www-ye/ego2robot_blog/blob/main/ego2robot/processors/calibration_ctrl.py): `ego2robot/processors/calibration_ctrl.py`

### RoboTwin Evaluation

- [Task Environments](https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/envs/_base_task.py): `evaluation/robotwin/envs/_base_task.py`
- [Simulation Cameras](https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/envs/camera/camera.py): `evaluation/robotwin/envs/camera/camera.py`
- [Robot Planning](https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/envs/robot/robot.py): `evaluation/robotwin/envs/robot/robot.py`
- [Motion Planner](https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/envs/robot/planner.py): `evaluation/robotwin/envs/robot/planner.py`
- [EEF Controller](https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/eef_control/controller.py): `evaluation/robotwin/eef_control/controller.py`
- [IK And FK Solver](https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/eef_control/mink_ik_solver.py): `evaluation/robotwin/eef_control/mink_ik_solver.py`
- [Policy Evaluation](https://github.com/www-ye/ego2robot_blog/blob/main/evaluation/robotwin/script/eval_policy.py): `evaluation/robotwin/script/eval_policy.py`

### Other components

- **Dataset User**
- **Policy Evaluator**

## Connections

- Dataset User → Pipeline Driver: runs (evidence: `README.md`)
- Pipeline Driver → Base Processor: orchestrates (evidence: `ego2robot/pipeline.py`)
- Pipeline Driver → EgoDex Adapter: selects source (evidence: `README.md`)
- Pipeline Driver → EgoVerse Adapter: selects source (evidence: `README.md`)
- Pipeline Driver → ViTRA Adapter: selects source (evidence: `README.md`)
- Pipeline Driver → Subtask Detection: runs first (evidence: `README.md`)
- Subtask Detection → Depth And Pose: feeds (evidence: `README.md`)
- Depth And Pose → Hand Reconstruction: feeds (evidence: `README.md`)
- Hand Reconstruction → Action Retargeting: feeds (evidence: `README.md`)
- Pipeline Driver → Action Retargeting: runs stage (evidence: `README.md`)
- Action Retargeting → Trajectory Smoothing: feeds (evidence: `README.md`)
- Trajectory Smoothing → Arm Segmentation: feeds (evidence: `README.md`)
- Arm Segmentation → Hand Inpainting: feeds (evidence: `README.md`)
- Hand Inpainting → Robot Rendering: feeds (evidence: `README.md`)
- Hand Reconstruction → AnyHand Model: uses (evidence: `ego2robot/processors/hand_processor.py`)
- Hand Reconstruction → WiLoR Model: uses (evidence: `ego2robot/processors/hand_processor.py`)
- Depth And Pose → Depth Anything: uses
- Robot Rendering → MuJoCo Twin: delegates to (evidence: `ego2robot/processors/robot_inpaint/processor.py`)
- MuJoCo Twin → Camera Calibration: uses (evidence: `ego2robot/processors/robot_inpaint/mujoco_twin_robot.py`)
- Dataset User → LeRobot Export: exports data (evidence: `README.md`)
- Policy Evaluator → Policy Evaluation: runs
- Policy Evaluation → Task Environments: evaluates in
- Task Environments → Simulation Cameras: uses (evidence: `evaluation/robotwin/envs/_base_task.py`)
- Task Environments → Robot Planning: uses (evidence: `evaluation/robotwin/envs/_base_task.py`)
- Robot Planning → Motion Planner: dispatches to (evidence: `evaluation/robotwin/envs/robot/robot.py`)
- EEF Controller → IK And FK Solver: uses (evidence: `evaluation/robotwin/eef_control/controller.py`)
