# unitreerobotics/avp_teleoperate architecture

> Architecture diagram of the GitHub repository unitreerobotics/avp_teleoperate, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/unitreerobotics/avp_teleoperate
- Repository: https://github.com/unitreerobotics/avp_teleoperate

## Overview

##### Project Type and Purpose
- This project implements teleoperation for Unitree humanoid robots using Apple Vision Pro. It interfaces between immersive VR controls and robot joint control (both arms and dexterous hands).
- It is a robotics teleoperation system that integrates real-time image streaming (from onboard cameras), inverse kinematics for arm control, and network-based communication between the operator’s device (Apple Vision Pro) and the robot’s onboard computers.

##### File Structure
- Key top-level directories include:
- assets – Contains URDF files, meshes, simulation assets for various robot configurations (G1, H1_2, Inspire Hand, Unitree Hand). These describe the robot’s physical structure.
- teleop – Holds all teleoperation components. Inside this, you have:
▪ image_server – Code for capturing and sending head camera images (runs on the robot’s PC2).
▪ open_television – Modules that capture wrist and hand data from the Apple Vision Pro device.
▪ robot_control – Includes inverse kinematics (robot_arm_ik.py) and modules for controlling arms and hands (robot_arm, robot_hand_inspire, robot_hand_unitree).
▪ utils – Supplemental tools such as data recorders, mathematical utilities, and filtering components.
▪ teleop_hand_and_arm.py – The startup script tying together the teleoperation system.
- Additional directories (hardware, requirements.txt) support integration with external hardware (camera mounts, STEP files) and manage dependencies.

##### Key Components
a. User/VR Interface:
- Apple Vision Pro aspects represented in “open_television”.
- Captures user’s head, wrist, and hand motion data through modules like tv_wrapper and television.py.
b. Image Acquisition and Streaming:
- “teleop/image_server” including both image_server.py (which reads data from cameras using OpenCV or RealSense SDK) and image_client.py to test connection.
- This demonstrates the pipeline for acquiring and streaming visual data.
c. Robot Control System:
- “teleop/robot_control” houses modules for controlling the robot’s limbs.
- robot_arm.py and robot_arm_ik.py: Compute inverse kinematics and implementing robot arm control.
- robot_hand_inspire.py and robot_hand_unitree.py: Specific implementations for controlling dexterous hand joints.
- These modules process user input (from Apple Vision Pro) and send commands to the robot’s onboard computing (PC2).
d. Data Utilities and Recording:
- “teleop/utils”: Contains modules for data recording (episode_writer.py) and filtering (weighted_moving_filter.py) that support real-time robot control and offline data analysis.
e. Physical and Hardware Interfaces:
- The “assets” directory holds the robot’s physical descriptions defined via URDF and CAD meshes.
- “hardware” directory contains STEP files and documentation for physical mounts.

##### Relationships and Data Flow
- The VR Interface (Apple Vision Pro via the open_television module) captures human motions and translates them into control commands.
- These commands are processed by the teleop_hand_and_arm.py startup script, which delegates tasks to:
- The robot_control modules (for inverse kinematics and joint motion commands).
- The image_server module, which streams visual feedback from head cameras to the user’s device over secured network channels.
- Data flows between the PC (running teleop controllers) and the robot’s onboard computing unit (PC2), likely over a local network (using protocols like WSS).
- The physical mapping from software commands to robot actuation is modeled using the assets (URDF/meshes) representing real hardware.

##### Important Architectural Patterns/Design Principles
- Modular Separation: Clearly defines modules for image streaming, motion data processing, robot control, and utilities. This improves maintainability and allows independent testing.
- Client-Server Architecture: The image_server on the robot (server) streams data to the client running on the host machine.
- Hardware Abstraction: URDF assets and hardware STEP files separate physical robot configurations from control algorithms.
- Real-Time Data Processing: Inverse kinematics and filtering modules ensure smooth, real-time teleoperation.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top-level VR Device
    VR["Apple Vision Pro"]:::software

    %% Subgraph: OpenTeleVision Module
    subgraph "OpenTeleVision Module"
        TVW["TV Wrapper"]:::software
        TV["Television"]:::software
    end

    %% Teleoperation Startup
    TS["Teleop Startup\n(teleop_hand_and_arm.py)"]:::software

    %% Subgraph: Image Streaming Modules
    subgraph "Image Streaming"
        IS["Image Server\n(image_server.py)"]:::software
        IC["Image Client\n(image_client.py)"]:::software
    end

    %% Subgraph: Robot Control Modules
    subgraph "Robot Control Modules"
        ARMIK["Arm IK\n(robot_arm_ik.py)"]:::software
        ARMCTRL["Arm Control\n(robot_arm.py)"]:::software
        HANDINSPIRE["Hand Control Inspire\n(robot_hand_inspire.py)"]:::software
        HANDUNITREE["Hand Control Unitree\n(robot_hand_unitree.py)"]:::software
    end

    %% Subgraph: Data Utilities
    subgraph "Data Utilities"
        EW["Episode Writer\n(episode_writer.py)"]:::software
        WMF["Weighted Moving Filter\n(weighted_moving_filter.py)"]:::software
        MT["Mat Tool\n(mat_tool.py)"]:::software
        RV["Rerun Visualizer\n(rerun_visualizer.py)"]:::software
    end

    %% Subgraph: Physical Robot Assets
    subgraph "Robot Assets"
        G1["G1 (Asset)"]:::hardware
        H12["H1_2 (Asset)"]:::hardware
        INSPIRE["Inspire Hand (Asset)"]:::hardware
        UNITREE["Unitree Hand (Asset)"]:::hardware
    end

    %% Hardware Interfaces
    HW["Hardware Interfaces"]:::hardware

    %% Connections - VR and OpenTeleVision
    VR -->|"MotionData"| TVW
    VR -->|"MotionData"| TV

    %% OpenTeleVision to Teleop Startup
    TVW -->|"SendCmds"| TS
    TV -->|"SendCmds"| TS

    %% Teleop Startup to Image Streaming and Robot Control
    TS -->|"ImageRequest"| IS
    TS -->|"ControlCmds"| ARMIK
    TS -->|"ControlCmds"| ARMCTRL
    TS -->|"ControlCmds"| HANDINSPIRE
    TS -->|"ControlCmds"| HANDUNITREE
    TS -->|"RecordData"| EW

    %% Image Streaming Pipeline
    IS -->|"ImageStream"| IC
    IC -->|"VisualFeedback"| VR

    %% Data Utilities support Robot Control
    WMF -->|"FilteredData"| ARMCTRL
    MT -->|"DataTool"| ARMIK
    RV -->|"DataViz"| HANDINSPIRE
    %% Also, TS passes data to Data Utilities (recording)
    TS -->|"LogData"| EW

    %% Robot Control to Physical Assets
    ARMIK -->|"Actuation"| G1
    ARMCTRL -->|"Actuation"| H12
    HANDINSPIRE -->|"Actuation"| INSPIRE
    HANDUNITREE -->|"Actuation"| UNITREE

    %% Physical Assets linked to Hardware Interfaces
    G1 -->|"AssembledWith"| HW
    H12 -->|"AssembledWith"| HW
    INSPIRE -->|"AssembledWith"| HW
    UNITREE -->|"AssembledWith"| HW

    %% Click Events for OpenTeleVision Modules
    click TVW "https://github.com/unitreerobotics/avp_teleoperate/blob/main/teleop/open_television/tv_wrapper.py"
    click TV "https://github.com/unitreerobotics/avp_teleoperate/blob/main/teleop/open_television/television.py"

    %% Click Event for Teleop Startup Script
    click TS "https://github.com/unitreerobotics/avp_teleoperate/blob/main/teleop/teleop_hand_and_arm.py"

    %% Click Events for Image Acquisition and Streaming
    click IS "https://github.com/unitreerobotics/avp_teleoperate/blob/main/teleop/image_server/image_server.py"
    click IC "https://github.com/unitreerobotics/avp_teleoperate/blob/main/teleop/image_server/image_client.py"

    %% Click Events for Robot Control Modules
    click ARMIK "https://github.com/unitreerobotics/avp_teleoperate/blob/main/teleop/robot_control/robot_arm_ik.py"
    click ARMCTRL "https://github.com/unitreerobotics/avp_teleoperate/blob/main/teleop/robot_control/robot_arm.py"
    click HANDINSPIRE "https://github.com/unitreerobotics/avp_teleoperate/blob/main/teleop/robot_control/robot_hand_inspire.py"
    click HANDUNITREE "https://github.com/unitreerobotics/avp_teleoperate/blob/main/teleop/robot_control/robot_hand_unitree.py"

    %% Click Events for Data Utilities
    click EW "https://github.com/unitreerobotics/avp_teleoperate/blob/main/teleop/utils/episode_writer.py"
    click WMF "https://github.com/unitreerobotics/avp_teleoperate/blob/main/teleop/utils/weighted_moving_filter.py"
    click MT "https://github.com/unitreerobotics/avp_teleoperate/blob/main/teleop/utils/mat_tool.py"
    click RV "https://github.com/unitreerobotics/avp_teleoperate/blob/main/teleop/utils/rerun_visualizer.py"

    %% Click Events for Physical Robot Assets
    click G1 "https://github.com/unitreerobotics/avp_teleoperate/tree/main/assets/g1"
    click H12 "https://github.com/unitreerobotics/avp_teleoperate/tree/main/assets/h1_2"
    click INSPIRE "https://github.com/unitreerobotics/avp_teleoperate/tree/main/assets/inspire_hand"
    click UNITREE "https://github.com/unitreerobotics/avp_teleoperate/tree/main/assets/unitree_hand"

    %% Click Events for Hardware Interfaces
    click HW "https://github.com/unitreerobotics/avp_teleoperate/tree/main/hardware"

    %% Styles
    classDef software fill:#cce5ff,stroke:#004085,stroke-width:2px,font-size:12px;
    classDef hardware fill:#d4edda,stroke:#155724,stroke-width:2px,font-size:12px;
```
