# hongzhebi/h_rdt architecture

> Architecture diagram of the GitHub repository hongzhebi/h_rdt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hongzhebi/h_rdt
- Repository: https://github.com/hongzhebi/h_rdt

## Overview

H-RDT trains diffusion-transformer manipulation policies by first learning behavioral priors from egocentric human hand/action data, then fine-tuning across robot embodiments, and deploys the resulting policy for robot control. The graph captures the documented EgoDx preprocessing and RoboTwin2 data branch, shared dataset/model training path, and real-robot inference path. Source excerpts are partial, so uncertain details of training internals and unsampled integration wiring are omitted.

Main entry points: [`train/train.py`](https://github.com/hongzhebi/h_rdt/blob/main/train/train.py) (Training loop), [`inference/real_example/inference_hrdt.py`](https://github.com/hongzhebi/h_rdt/blob/main/inference/real_example/inference_hrdt.py) (Robot inference loop), [`datasets/pretrain/egodex_dataset.py`](https://github.com/hongzhebi/h_rdt/blob/main/datasets/pretrain/egodex_dataset.py) (Human dataset), [`datasets/dataset.py`](https://github.com/hongzhebi/h_rdt/blob/main/datasets/dataset.py) (Dataset selection)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_precompute["48D action conversion"]
  node_egodex_dataset["Human dataset<br/>[egodex_dataset.py]"]
  node_robotwin_dataset["Robot dataset"]
  node_language_prep["Language embeddings"]
  node_dataset_router["Dataset selection<br/>[dataset.py]"]
  node_batching["Batch collation<br/>[dataset.py]"]
end

subgraph group_learning["Policy learning"]
  node_entry["Training entry<br/>[main.py]"]
  node_trainer["Training loop<br/>[train.py]"]
end

subgraph group_inference["Robot deployment"]
  node_real_operator["Robot operator<br/>[ros_operator.py]"]
  node_real_entry["Robot inference loop<br/>[inference_hrdt.py]"]
  node_real_policy["Inference adapter<br/>[model_inference.py]"]
  node_video["Video recording<br/>[video_recorder.py]"]
  node_robotwin_deploy["RoboTwin deployment<br/>[deploy_policy.py]"]
end

subgraph group_core["Model components"]
  node_vision["Vision backbone<br/>[dinosiglip_vit.py]"]
  node_language_encoder["Text encoder<br/>[t5_encoder.py]"]
  node_runner["Policy runner<br/>[hrdt_runner.py]"]
  node_transformer["Diffusion transformer<br/>[model.py]"]
end

node_researcher(("Policy engineer"))
node_egodex[("EgoDx human data")]
node_robotwin[("RoboTwin2 episodes")]
node_robot(("Bimanual robot"))

node_researcher -->|"starts training"| node_entry
node_egodex -->|"provides episodes"| node_precompute
node_precompute -->|"stores actions"| node_egodex_dataset
node_egodex -->|"provides samples"| node_egodex_dataset
node_robotwin -->|"provides episodes"| node_robotwin_dataset
node_egodex_dataset -->|"supplies samples"| node_dataset_router
node_robotwin_dataset -->|"supplies samples"| node_dataset_router
node_language_prep -->|"provides embeddings"| node_egodex_dataset
node_dataset_router -->|"returns examples"| node_batching
node_entry -->|"calls"| node_trainer
node_trainer -->|"constructs"| node_dataset_router
node_trainer -->|"batches data"| node_batching
node_trainer -->|"encodes images"| node_vision
node_trainer -->|"trains"| node_runner
node_runner -->|"runs model"| node_transformer
node_real_operator -->|"supplies observations"| node_real_entry
node_real_entry -->|"updates and queries"| node_real_policy
node_real_policy -->|"encodes views"| node_vision
node_real_policy -->|"predicts actions"| node_runner
node_real_entry -->|"publishes actions"| node_real_operator
node_real_operator -->|"controls"| node_robot
node_real_entry -->|"records views"| node_video
node_researcher -.->|"launches deployment"| node_robotwin_deploy
node_robotwin_deploy -.->|"loads policy"| node_runner

click node_precompute "https://github.com/hongzhebi/h_rdt/blob/main/datasets/pretrain/precompute_48d_actions.py"
click node_egodex_dataset "https://github.com/hongzhebi/h_rdt/blob/main/datasets/pretrain/egodex_dataset.py"
click node_robotwin_dataset "https://github.com/hongzhebi/h_rdt/blob/main/datasets/robotwin2/robotwin_agilex_dataset.py"
click node_language_prep "https://github.com/hongzhebi/h_rdt/blob/main/datasets/pretrain/encode_lang_batch.py"
click node_dataset_router "https://github.com/hongzhebi/h_rdt/blob/main/datasets/dataset.py"
click node_batching "https://github.com/hongzhebi/h_rdt/blob/main/datasets/dataset.py"
click node_entry "https://github.com/hongzhebi/h_rdt/blob/main/main.py"
click node_trainer "https://github.com/hongzhebi/h_rdt/blob/main/train/train.py"
click node_vision "https://github.com/hongzhebi/h_rdt/blob/main/models/encoder/dinosiglip_vit.py"
click node_language_encoder "https://github.com/hongzhebi/h_rdt/blob/main/models/encoder/t5_encoder.py"
click node_runner "https://github.com/hongzhebi/h_rdt/blob/main/models/hrdt_runner.py"
click node_transformer "https://github.com/hongzhebi/h_rdt/blob/main/models/hrdt/model.py"
click node_real_operator "https://github.com/hongzhebi/h_rdt/blob/main/inference/real_example/ros_operator.py"
click node_real_entry "https://github.com/hongzhebi/h_rdt/blob/main/inference/real_example/inference_hrdt.py"
click node_real_policy "https://github.com/hongzhebi/h_rdt/blob/main/inference/real_example/model_inference.py"
click node_video "https://github.com/hongzhebi/h_rdt/blob/main/inference/real_example/video_recorder.py"
click node_robotwin_deploy "https://github.com/hongzhebi/h_rdt/blob/main/inference/robotwin2_example/H-RDT/deploy_policy.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_precompute,node_egodex_dataset,node_robotwin_dataset,node_language_prep,node_dataset_router,node_batching toneBlue
class node_entry,node_trainer,node_egodex,node_robotwin toneAmber
class node_real_operator,node_real_entry,node_real_policy,node_video,node_robotwin_deploy toneMint
class node_vision,node_language_encoder,node_runner,node_transformer toneRose
class node_researcher,node_robot toneIndigo
```

## Components

### Data preparation

- [48D action conversion](https://github.com/hongzhebi/h_rdt/blob/main/datasets/pretrain/precompute_48d_actions.py): `datasets/pretrain/precompute_48d_actions.py`
- [Human dataset](https://github.com/hongzhebi/h_rdt/blob/main/datasets/pretrain/egodex_dataset.py): `datasets/pretrain/egodex_dataset.py`
- [Robot dataset](https://github.com/hongzhebi/h_rdt/blob/main/datasets/robotwin2/robotwin_agilex_dataset.py): `datasets/robotwin2/robotwin_agilex_dataset.py`
- [Language embeddings](https://github.com/hongzhebi/h_rdt/blob/main/datasets/pretrain/encode_lang_batch.py): `datasets/pretrain/encode_lang_batch.py`
- [Dataset selection](https://github.com/hongzhebi/h_rdt/blob/main/datasets/dataset.py): `datasets/dataset.py`
- [Batch collation](https://github.com/hongzhebi/h_rdt/blob/main/datasets/dataset.py): `datasets/dataset.py`

### Policy learning

- [Training entry](https://github.com/hongzhebi/h_rdt/blob/main/main.py): `main.py`
- [Training loop](https://github.com/hongzhebi/h_rdt/blob/main/train/train.py): `train/train.py`

### Robot deployment

- [Robot operator](https://github.com/hongzhebi/h_rdt/blob/main/inference/real_example/ros_operator.py): `inference/real_example/ros_operator.py`
- [Robot inference loop](https://github.com/hongzhebi/h_rdt/blob/main/inference/real_example/inference_hrdt.py): `inference/real_example/inference_hrdt.py`
- [Inference adapter](https://github.com/hongzhebi/h_rdt/blob/main/inference/real_example/model_inference.py): `inference/real_example/model_inference.py`
- [Video recording](https://github.com/hongzhebi/h_rdt/blob/main/inference/real_example/video_recorder.py): `inference/real_example/video_recorder.py`
- [RoboTwin deployment](https://github.com/hongzhebi/h_rdt/blob/main/inference/robotwin2_example/H-RDT/deploy_policy.py): `inference/robotwin2_example/H-RDT/deploy_policy.py`

### Model components

- [Vision backbone](https://github.com/hongzhebi/h_rdt/blob/main/models/encoder/dinosiglip_vit.py): `models/encoder/dinosiglip_vit.py`
- [Text encoder](https://github.com/hongzhebi/h_rdt/blob/main/models/encoder/t5_encoder.py): `models/encoder/t5_encoder.py`
- [Policy runner](https://github.com/hongzhebi/h_rdt/blob/main/models/hrdt_runner.py): `models/hrdt_runner.py`
- [Diffusion transformer](https://github.com/hongzhebi/h_rdt/blob/main/models/hrdt/model.py): `models/hrdt/model.py`

### Other components

- **Policy engineer**
- **EgoDx human data**
- **RoboTwin2 episodes**
- **Bimanual robot**

## Connections

- Policy engineer → Training entry: starts training
- EgoDx human data → 48D action conversion: provides episodes
- 48D action conversion → Human dataset: stores actions
- EgoDx human data → Human dataset: provides samples
- RoboTwin2 episodes → Robot dataset: provides episodes
- Human dataset → Dataset selection: supplies samples
- Robot dataset → Dataset selection: supplies samples
- Language embeddings → Human dataset: provides embeddings
- Dataset selection → Batch collation: returns examples
- Training entry → Training loop: calls
- Training loop → Dataset selection: constructs
- Training loop → Batch collation: batches data
- Training loop → Vision backbone: encodes images
- Training loop → Policy runner: trains
- Policy runner → Diffusion transformer: runs model
- Robot operator → Robot inference loop: supplies observations
- Robot inference loop → Inference adapter: updates and queries
- Inference adapter → Vision backbone: encodes views
- Inference adapter → Policy runner: predicts actions
- Robot inference loop → Robot operator: publishes actions
- Robot operator → Bimanual robot: controls
- Robot inference loop → Video recording: records views
- Policy engineer → RoboTwin deployment: launches deployment
- RoboTwin deployment → Policy runner: loads policy