How unitreerobotics/unitree_il_lerobot works
Main Components
• Data Collection & Conversion:
• Raw Data Input: Represent raw JSON datasets obtained from Unitree robots (e.g., from the avp_teleoperate tool).
• Sorting & Renaming: Include the process executed by the script in unitree_lerobot/utils/sort_and_rename_folders.py.
• Data Conversion: Highlight the conversion process (script: convert_unitree_json_to_lerobot.py) that transforms raw Unitree JSON data into LeRobot’s expected format.
• Training Module (LeRobot Integration):
• LeRobot Framework: Represent that the project builds on the LeRobot framework. Show that this subsystem includes training scripts and a dedicated “lerobot” directory (unitree_lerobot/lerobot).
• Training Scripts: Illustrate training entry points for different policies (act, diffusion, pi0) as separate nodes or branches within the training component.
• Visualization: Reference the visualization script (visualize_dataset.py) which is used for dataset inspection.
• Evaluation / Real-World Inference:
• Evaluation Module: Represent the “eval_robot” directory that contains the evaluation pipeline for real robot inference.
• Sub-components:
▪ Image Server: Represent the image_server module that includes an image_client and image_server, facilitating the visual feedback or camera feed.
▪ Robot Control: Capture the robot_control module noting its responsibility to manage commands sent to the robot actuators (e.g., robot_arm.py, robot_hand_unitree.py).
• External Dependencies:
• LeRobot Framework: Clearly note the dependency on the LeRobot ecosystem and its role in training and evaluation.
• unitree_sdk2_python: Represent the additional dependency for DDS communication needed for real robot interaction.
• External Data Sources: Link to external repositories (e.g., the Hugging Face Unitree dataset and the avp_teleoperate repository).
Relationships and Interactions
• Data Flow:
• Converted data is either stored locally or pushed to Hugging Face Hub (indicated in the convert_unitree_json_to_lerobot.py command) before being used in training.
• Once training is complete, the trained model checkpoints are fed into the evaluation module to test them on either real hardware or directly on the dataset.
• Control Flow:
• The training pipeline interacts with the LeRobot framework components while model evaluation uses dedicated run scripts to interact with the robot’s control APIs (DDS through unitree_sdk2_python) and image server for feedback.
Architectural Patterns & Design Principles
• Separation of Concerns: Clearly distinguish the modules (data conversion, model training, and evaluation on real hardware). Each directory in the repository corresponds to a logical component.
• Modularity: Illustrate that utility functions (in the utils folder) are decoupled from the main training and evaluation workflows.
• Integration Points: Identify integration with external frameworks (LeRobot and unitree_sdk2_python) and external data repositories (Hugging Face).
Relevant Technologies and Frameworks
• LeRobot (for training, dataset handling, and visualization).
• Python and Conda (used for environment and dependency management).
• DDS Communication: Through unitree_sdk2_python for robot control and telemetry.
• External APIs: Hugging Face Hub for dataset hosting and model repository.