How hongzhebi/h_rdt works
H-RDT trains diffusion-transformer manipulation policies by first learning behavioral priors from egocentric human hand/action data, then fine-tuning across robot embodiments, and deploys the resulting policy for robot control. The graph captures the documented EgoDx preprocessing and RoboTwin2 data branch, shared dataset/model training path, and real-robot inference path. Source excerpts are partial, so uncertain details of training internals and unsampled integration wiring are omitted.
Main entry points: train.py Training loop · inference_hrdt.py Robot inference loop · egodex_dataset.py Human dataset · dataset.py Dataset selection
Components 21
Data preparation
- 48D action conversion
datasets/pretrain/precompute_48d_actions.py - Human dataset
datasets/pretrain/egodex_dataset.py - Robot dataset
datasets/robotwin2/robotwin_agilex_dataset.py - Language embeddings
datasets/pretrain/encode_lang_batch.py - Dataset selection
datasets/dataset.py - Batch collation
datasets/dataset.py
Policy learning
- Training entry
main.py - Training loop
train/train.py
Robot deployment
- Robot operator
inference/real_example/ros_operator.py - Robot inference loop
inference/real_example/inference_hrdt.py - Inference adapter
inference/real_example/model_inference.py - Video recording
inference/real_example/video_recorder.py - RoboTwin deployment
inference/robotwin2_example/H-RDT/deploy_policy.py
Model components
- Vision backbone
models/encoder/dinosiglip_vit.py - Text encoder
models/encoder/t5_encoder.py - Policy runner
models/hrdt_runner.py - Diffusion transformer
models/hrdt/model.py
Other components
- Policy engineer
- EgoDx human data
- RoboTwin2 episodes
- Bimanual robot
Connections 24
- Policy engineer → Training entry: starts training
- EgoDx human data → 48D action conversion: provides episodes
- 48D action conversion → Human dataset: stores actions
- EgoDx human data → Human dataset: provides samples
- RoboTwin2 episodes → Robot dataset: provides episodes
- Human dataset → Dataset selection: supplies samples
- Robot dataset → Dataset selection: supplies samples
- Language embeddings → Human dataset: provides embeddings
- Dataset selection → Batch collation: returns examples
- Training entry → Training loop: calls
- Training loop → Dataset selection: constructs
- Training loop → Batch collation: batches data
- Training loop → Vision backbone: encodes images
- Training loop → Policy runner: trains
- Policy runner → Diffusion transformer: runs model
- Robot operator → Robot inference loop: supplies observations
- Robot inference loop → Inference adapter: updates and queries
- Inference adapter → Vision backbone: encodes views
- Inference adapter → Policy runner: predicts actions
- Robot inference loop → Robot operator: publishes actions
- Robot operator → Bimanual robot: controls
- Robot inference loop → Video recording: records views
- Policy engineer → RoboTwin deployment: launches deployment
- RoboTwin deployment → Policy runner: loads policy