How unitreerobotics/avp_teleoperate works
Project Type and Purpose
• This project implements teleoperation for Unitree humanoid robots using Apple Vision Pro. It interfaces between immersive VR controls and robot joint control (both arms and dexterous hands).
• It is a robotics teleoperation system that integrates real-time image streaming (from onboard cameras), inverse kinematics for arm control, and network-based communication between the operator’s device (Apple Vision Pro) and the robot’s onboard computers.
File Structure
• Key top-level directories include:
• assets – Contains URDF files, meshes, simulation assets for various robot configurations (G1, H1_2, Inspire Hand, Unitree Hand). These describe the robot’s physical structure.
• teleop – Holds all teleoperation components. Inside this, you have:
▪ image_server – Code for capturing and sending head camera images (runs on the robot’s PC2).
▪ open_television – Modules that capture wrist and hand data from the Apple Vision Pro device.
▪ robot_control – Includes inverse kinematics (robot_arm_ik.py) and modules for controlling arms and hands (robot_arm, robot_hand_inspire, robot_hand_unitree).
▪ utils – Supplemental tools such as data recorders, mathematical utilities, and filtering components.
▪ teleop_hand_and_arm.py – The startup script tying together the teleoperation system.
• Additional directories (hardware, requirements.txt) support integration with external hardware (camera mounts, STEP files) and manage dependencies.
Key Components
a. User/VR Interface:
• Apple Vision Pro aspects represented in “open_television”.
• Captures user’s head, wrist, and hand motion data through modules like tv_wrapper and television.py.
b. Image Acquisition and Streaming:
• “teleop/image_server” including both image_server.py (which reads data from cameras using OpenCV or RealSense SDK) and image_client.py to test connection.
• This demonstrates the pipeline for acquiring and streaming visual data.
c. Robot Control System:
• “teleop/robot_control” houses modules for controlling the robot’s limbs.
• robot_arm.py and robot_arm_ik.py: Compute inverse kinematics and implementing robot arm control.
• robot_hand_inspire.py and robot_hand_unitree.py: Specific implementations for controlling dexterous hand joints.
• These modules process user input (from Apple Vision Pro) and send commands to the robot’s onboard computing (PC2).
d. Data Utilities and Recording:
• “teleop/utils”: Contains modules for data recording (episode_writer.py) and filtering (weighted_moving_filter.py) that support real-time robot control and offline data analysis.
e. Physical and Hardware Interfaces:
• The “assets” directory holds the robot’s physical descriptions defined via URDF and CAD meshes.
• “hardware” directory contains STEP files and documentation for physical mounts.
Relationships and Data Flow
• The VR Interface (Apple Vision Pro via the open_television module) captures human motions and translates them into control commands.
• These commands are processed by the teleop_hand_and_arm.py startup script, which delegates tasks to:
• The robot_control modules (for inverse kinematics and joint motion commands).
• The image_server module, which streams visual feedback from head cameras to the user’s device over secured network channels.
• Data flows between the PC (running teleop controllers) and the robot’s onboard computing unit (PC2), likely over a local network (using protocols like WSS).
• The physical mapping from software commands to robot actuation is modeled using the assets (URDF/meshes) representing real hardware.
Important Architectural Patterns/Design Principles
• Modular Separation: Clearly defines modules for image streaming, motion data processing, robot control, and utilities. This improves maintainability and allows independent testing.
• Client-Server Architecture: The image_server on the robot (server) streams data to the client running on the host machine.
• Hardware Abstraction: URDF assets and hardware STEP files separate physical robot configurations from control algorithms.
• Real-Time Data Processing: Inverse kinematics and filtering modules ensure smooth, real-time teleoperation.