How mondo-robotics/dit4dit works
DiT4DiT is a vision-action model for tabletop and humanoid manipulation. Its principal workflow loads LeRobot demonstrations, transforms video/state/action data, trains a video-and-action model, then serves predictions for robot control. The repository also includes real-G1 sensing and control, model-serving transports, and a bundled Unitree SDK. The graph connects only relationships supported by the README or SOURCE INDEX references; some subsystem links use the README where implementation excerpts were unavailable.
Main entry points: datasets.py LeRobot datasets · DiT4DiT.py DiT4DiT framework · run_dit4dit_inference_policy_loop.py G1 inference loop · train.py Training entry
Components 26
Data ingestion
- LeRobot datasets
DiT4DiT/dataloader/gr00t_lerobot/datasets.py - Data schema
DiT4DiT/dataloader/gr00t_lerobot/schema.py - Embodiment tags
DiT4DiT/dataloader/gr00t_lerobot/embodiment_tags.py - Transform base
DiT4DiT/dataloader/gr00t_lerobot/transform/base.py - State/action transforms
DiT4DiT/dataloader/gr00t_lerobot/transform/state_action.py - Video transforms
DiT4DiT/dataloader/gr00t_lerobot/transform/video.py - Video loading
DiT4DiT/dataloader/gr00t_lerobot/video.py
Model and training
- Training entry
DiT4DiT/training/train.py - DiT4DiT framework
DiT4DiT/model/framework/DiT4DiT.py - Framework base
DiT4DiT/model/framework/base_framework.py - Shared model tools
DiT4DiT/model/framework/share_tools.py - Cosmos backbone
DiT4DiT/model/modules/vlm/Cosmos25.py - Action DiT
DiT4DiT/model/modules/action_model/ActionDiT.py - Flow matching head
DiT4DiT/model/modules/action_model/flow_matching_head/cross_attention_dit.py - Model utilities
DiT4DiT/model/tools.py - Trainer utilities
DiT4DiT/training/trainer_utils/trainer_tools.py - Config tracking
DiT4DiT/training/trainer_utils/config_tracker.py
Inference serving
- ZMQ policy server
deployment/model_server/server_policy_zmq.py
Robot control
- Inference service
decoupled_wbc/gr00t_wbc/control/utils/service.py - G1 inference loop
decoupled_wbc/gr00t_wbc/control/main/teleop/run_dit4dit_inference_policy_loop.py - Composed camera
decoupled_wbc/gr00t_wbc/control/sensor/composed_camera.py
Unitree SDK
- RPC client
decoupled_wbc/external_dependencies/unitree_sdk2_python/unitree_sdk2py/rpc/client.py - RPC internals
decoupled_wbc/external_dependencies/unitree_sdk2_python/unitree_sdk2py/rpc/internal.py - Sport client
decoupled_wbc/external_dependencies/unitree_sdk2_python/unitree_sdk2py/go2/sport/sport_client.py
Other components
- Robot operator
- Unitree G1
Connections 30
- Training entry → LeRobot datasets: loads batches
DiT4DiT/training/train.py - Training entry → DiT4DiT framework: trains model
DiT4DiT/training/train.py - Training entry → Trainer utilities: uses utilities
DiT4DiT/training/train.py - Training entry → Config tracking: tracks config
DiT4DiT/training/train.py - LeRobot datasets → Video loading: loads video
DiT4DiT/dataloader/gr00t_lerobot/datasets.py - LeRobot datasets → Data schema: uses schema
DiT4DiT/dataloader/gr00t_lerobot/datasets.py - LeRobot datasets → Embodiment tags: uses tags
DiT4DiT/dataloader/gr00t_lerobot/datasets.py - LeRobot datasets → Transform base: uses transforms
DiT4DiT/dataloader/gr00t_lerobot/datasets.py - Transform base → Data schema: uses schema
DiT4DiT/dataloader/gr00t_lerobot/transform/base.py - State/action transforms → Transform base: extends base
DiT4DiT/dataloader/gr00t_lerobot/transform/state_action.py - State/action transforms → Data schema: uses schema
DiT4DiT/dataloader/gr00t_lerobot/transform/state_action.py - DiT4DiT framework → Framework base: extends framework
DiT4DiT/model/framework/DiT4DiT.py - DiT4DiT framework → Cosmos backbone: uses backbone
DiT4DiT/model/framework/DiT4DiT.py - DiT4DiT framework → Action DiT: uses action model
DiT4DiT/model/framework/DiT4DiT.py - DiT4DiT framework → Trainer utilities: uses utilities
DiT4DiT/model/framework/DiT4DiT.py - Framework base → Shared model tools: uses shared tools
DiT4DiT/model/framework/base_framework.py - Framework base → Model utilities: uses utilities
DiT4DiT/model/framework/base_framework.py - Action DiT → Flow matching head: uses head
DiT4DiT/model/modules/action_model/ActionDiT.py - Action DiT → Model utilities: uses utilities
DiT4DiT/model/framework/DiT4DiT.py - Robot operator → Training entry: trains policy
README.md - ZMQ policy server → Framework base: loads framework
deployment/model_server/server_policy_zmq.py - ZMQ policy server → Shared model tools: uses shared tools
deployment/model_server/server_policy_zmq.py - Robot operator → G1 inference loop: starts control
- G1 inference loop → Inference service: requests inference
decoupled_wbc/gr00t_wbc/control/main/teleop/run_dit4dit_inference_policy_loop.py - G1 inference loop → Composed camera: reads images
decoupled_wbc/gr00t_wbc/control/main/teleop/run_dit4dit_inference_policy_loop.py - G1 inference loop → ZMQ policy server: requests actions
README.md - G1 inference loop → Unitree G1: publishes actions
README.md - Sport client → RPC client: calls RPC
decoupled_wbc/external_dependencies/unitree_sdk2_python/unitree_sdk2py/go2/sport/sport_client.py - RPC client → RPC internals: uses internals
decoupled_wbc/external_dependencies/unitree_sdk2_python/unitree_sdk2py/rpc/client.py - Sport client → Unitree G1: controls robot