# wuphilipp/gello_software architecture

> Architecture diagram of the GitHub repository wuphilipp/gello_software, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wuphilipp/gello_software
- Repository: https://github.com/wuphilipp/gello_software

## Overview

GELLO is a robot-manipulator teleoperation framework. Its main loop turns a leader device’s input into joint commands for a configured real or simulated robot, while RobotEnv collects robot and camera observations. The repository also contains simulation tasks, demonstration-data conversion, standalone FACTR gravity compensation, and a ROS 2 Franka FR3 implementation. The README documents YAML-based and non-YAM launches and supported robot families; unsampled subsystems are represented at their existing directory paths, with links omitted where source evidence does not establish a call.

Main entry points: [`gello/env.py`](https://github.com/wuphilipp/gello_software/blob/main/gello/env.py) (Robot environment), [`gello/utils/launch_utils.py`](https://github.com/wuphilipp/gello_software/blob/main/gello/utils/launch_utils.py) (Launch orchestration), [`gello/factr/gravity_compensation.py`](https://github.com/wuphilipp/gello_software/blob/main/gello/factr/gravity_compensation.py) (FACTR gravity compensation), [`gello/cameras/camera.py`](https://github.com/wuphilipp/gello_software/blob/main/gello/cameras/camera.py) (Camera interface)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_teleop["Teleoperation"]
  node_launch["Launch orchestration<br/>[launch_utils.py]"]
  node_control_loop["Control loop<br/>[control_utils.py]"]
  node_agent["Agent interface<br/>[agent.py]"]
  node_gello_input["GELLO agent<br/>[gello_agent.py]"]
  node_spacemouse["SpaceMouse agent"]
  node_quest["Quest agent<br/>[quest_agent.py]"]
end

subgraph group_robot_io["Robot and Sensors"]
  node_robot_env["Robot environment<br/>[env.py]"]
  node_robot_protocol["Robot interface<br/>[robot.py]"]
  node_robot_adapters["Robot drivers"]
  node_dynamixel["Dynamixel driver<br/>[driver.py]"]
  node_camera["Camera interface<br/>[camera.py]"]
  node_realsense["RealSense camera"]
  node_robot_rpc["Robot ZMQ server/client<br/>[robot_node.py]"]
  node_camera_rpc["Camera ZMQ node<br/>[camera_node.py]"]
end

subgraph group_simulation_data["Simulation and Data"]
  node_dm_arms["Simulation arms<br/>[manipulator.py]"]
  node_dm_tasks["Manipulation tasks"]
  node_dm_arenas["Simulation arenas"]
  node_sim_robot["Mujoco robot<br/>[sim_robot.py]"]
  node_demo_conversion["Demo conversion<br/>[demo_to_gdict.py]"]
  node_conversion_utils["Conversion utilities"]
  node_mjcf_utils["MJCF utilities<br/>[mjcf_utils.py]"]
end

subgraph group_factr["FACTR Compensation"]
  node_factr_app["FACTR gravity compensation"]
  node_factr_agent["FACTR YAM agent"]
end

subgraph group_ros2["ROS 2 Franka"]
  node_ros_state["GELLO state publisher<br/>[gello_publisher.py]"]
  node_ros_hardware["Franka GELLO hardware<br/>[gello_hardware.py]"]
  node_ros_controller["Joint impedance controller"]
  node_ros_motion["Motion generator"]
  node_franka_gripper["Franka gripper client"]
  node_robotiq_gripper["Robotiq gripper client"]
end

node_operator(("Operator"))
node_robot_arm["Robot arm"]
node_camera_device["Camera device"]
node_franka_hardware["Franka FR3"]

node_operator -->|"starts"| node_launch
node_launch -->|"runs"| node_control_loop
node_control_loop -->|"requests action"| node_agent
node_gello_input -->|"implements"| node_agent
node_spacemouse -->|"implements"| node_agent
node_quest -->|"implements"| node_agent
node_control_loop -->|"steps"| node_robot_env
node_robot_env -->|"commands and reads"| node_robot_protocol
node_robot_env -->|"reads frames"| node_camera
node_realsense -->|"implements"| node_camera
node_robot_adapters -->|"implement"| node_robot_protocol
node_robot_adapters -->|"uses"| node_dynamixel
node_robot_rpc -->|"dispatches methods"| node_robot_protocol
node_camera_rpc -->|"uses"| node_camera
node_launch -->|"configures"| node_robot_rpc
node_launch -->|"initializes"| node_dynamixel
node_robot_env -.->|"commands via adapter"| node_robot_arm
node_camera -.->|"reads from"| node_camera_device
node_dm_tasks -->|"uses"| node_dm_arms
node_dm_tasks -->|"uses"| node_dm_arenas
node_dm_arms -->|"uses MJCF helpers"| node_mjcf_utils
node_demo_conversion -->|"uses"| node_conversion_utils
node_factr_app -->|"uses"| node_dynamixel
node_factr_app -.->|"optionally connects"| node_robot_rpc
node_factr_app -.->|"optionally creates"| node_robot_env
node_factr_agent -->|"uses"| node_factr_app
node_ros_state -.->|"uses hardware interface"| node_ros_hardware
node_ros_hardware -.->|"interfaces with"| node_franka_hardware
node_ros_controller -->|"uses"| node_ros_motion
node_franka_gripper -.->|"controls gripper"| node_franka_hardware
node_robotiq_gripper -.->|"controls gripper"| node_franka_hardware

click node_launch "https://github.com/wuphilipp/gello_software/blob/main/gello/utils/launch_utils.py"
click node_control_loop "https://github.com/wuphilipp/gello_software/blob/main/gello/utils/control_utils.py"
click node_agent "https://github.com/wuphilipp/gello_software/blob/main/gello/agents/agent.py"
click node_gello_input "https://github.com/wuphilipp/gello_software/blob/main/gello/agents/gello_agent.py"
click node_spacemouse "https://github.com/wuphilipp/gello_software/blob/main/gello/agents/spacemouse_agent.py"
click node_quest "https://github.com/wuphilipp/gello_software/blob/main/gello/agents/quest_agent.py"
click node_robot_env "https://github.com/wuphilipp/gello_software/blob/main/gello/env.py"
click node_robot_protocol "https://github.com/wuphilipp/gello_software/blob/main/gello/robots/robot.py"
click node_robot_adapters "https://github.com/wuphilipp/gello_software/tree/main/gello/robots"
click node_dynamixel "https://github.com/wuphilipp/gello_software/blob/main/gello/dynamixel/driver.py"
click node_camera "https://github.com/wuphilipp/gello_software/blob/main/gello/cameras/camera.py"
click node_realsense "https://github.com/wuphilipp/gello_software/blob/main/gello/cameras/realsense_camera.py"
click node_robot_rpc "https://github.com/wuphilipp/gello_software/blob/main/gello/zmq_core/robot_node.py"
click node_camera_rpc "https://github.com/wuphilipp/gello_software/blob/main/gello/zmq_core/camera_node.py"
click node_dm_arms "https://github.com/wuphilipp/gello_software/blob/main/gello/dm_control_tasks/arms/manipulator.py"
click node_dm_tasks "https://github.com/wuphilipp/gello_software/tree/main/gello/dm_control_tasks/manipulation/tasks"
click node_dm_arenas "https://github.com/wuphilipp/gello_software/tree/main/gello/dm_control_tasks/manipulation/arenas"
click node_sim_robot "https://github.com/wuphilipp/gello_software/blob/main/gello/robots/sim_robot.py"
click node_demo_conversion "https://github.com/wuphilipp/gello_software/blob/main/gello/data_utils/demo_to_gdict.py"
click node_conversion_utils "https://github.com/wuphilipp/gello_software/blob/main/gello/data_utils/conversion_utils.py"
click node_mjcf_utils "https://github.com/wuphilipp/gello_software/blob/main/gello/dm_control_tasks/mjcf_utils.py"
click node_factr_app "https://github.com/wuphilipp/gello_software/blob/main/gello/factr/gravity_compensation.py"
click node_factr_agent "https://github.com/wuphilipp/gello_software/blob/main/gello/factr/yam_gello_factr_agent.py"
click node_ros_state "https://github.com/wuphilipp/gello_software/blob/main/ros2/src/franka_gello_state_publisher/franka_gello_state_publisher/gello_publisher.py"
click node_ros_hardware "https://github.com/wuphilipp/gello_software/blob/main/ros2/src/franka_gello_state_publisher/franka_gello_state_publisher/gello_hardware.py"
click node_ros_controller "https://github.com/wuphilipp/gello_software/blob/main/ros2/src/franka_fr3_arm_controllers/src/joint_impedance_controller.cpp"
click node_ros_motion "https://github.com/wuphilipp/gello_software/blob/main/ros2/src/franka_fr3_arm_controllers/src/motion_generator.cpp"
click node_franka_gripper "https://github.com/wuphilipp/gello_software/blob/main/ros2/src/franka_gripper_manager/franka_gripper_manager/franka_gripper_client.py"
click node_robotiq_gripper "https://github.com/wuphilipp/gello_software/blob/main/ros2/src/franka_gripper_manager/franka_gripper_manager/robotiq_gripper_client.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_launch,node_control_loop,node_agent,node_gello_input,node_spacemouse,node_quest toneBlue
class node_robot_env,node_robot_protocol,node_robot_adapters,node_dynamixel,node_camera,node_realsense,node_robot_rpc,node_camera_rpc toneAmber
class node_dm_arms,node_dm_tasks,node_dm_arenas,node_sim_robot,node_demo_conversion,node_conversion_utils,node_mjcf_utils toneMint
class node_factr_app,node_factr_agent toneRose
class node_ros_state,node_ros_hardware,node_ros_controller,node_ros_motion,node_franka_gripper,node_robotiq_gripper,node_operator,node_robot_arm,node_camera_device,node_franka_hardware toneIndigo
```

## Components

### Teleoperation

- [Launch orchestration](https://github.com/wuphilipp/gello_software/blob/main/gello/utils/launch_utils.py): `gello/utils/launch_utils.py`
- [Control loop](https://github.com/wuphilipp/gello_software/blob/main/gello/utils/control_utils.py): `gello/utils/control_utils.py`
- [Agent interface](https://github.com/wuphilipp/gello_software/blob/main/gello/agents/agent.py): `gello/agents/agent.py`
- [GELLO agent](https://github.com/wuphilipp/gello_software/blob/main/gello/agents/gello_agent.py): `gello/agents/gello_agent.py`
- [SpaceMouse agent](https://github.com/wuphilipp/gello_software/blob/main/gello/agents/spacemouse_agent.py): `gello/agents/spacemouse_agent.py`
- [Quest agent](https://github.com/wuphilipp/gello_software/blob/main/gello/agents/quest_agent.py): `gello/agents/quest_agent.py`

### Robot and Sensors

- [Robot environment](https://github.com/wuphilipp/gello_software/blob/main/gello/env.py): `gello/env.py`
- [Robot interface](https://github.com/wuphilipp/gello_software/blob/main/gello/robots/robot.py): `gello/robots/robot.py`
- [Robot drivers](https://github.com/wuphilipp/gello_software/tree/main/gello/robots): `gello/robots`
- [Dynamixel driver](https://github.com/wuphilipp/gello_software/blob/main/gello/dynamixel/driver.py): `gello/dynamixel/driver.py`
- [Camera interface](https://github.com/wuphilipp/gello_software/blob/main/gello/cameras/camera.py): `gello/cameras/camera.py`
- [RealSense camera](https://github.com/wuphilipp/gello_software/blob/main/gello/cameras/realsense_camera.py): `gello/cameras/realsense_camera.py`
- [Robot ZMQ server/client](https://github.com/wuphilipp/gello_software/blob/main/gello/zmq_core/robot_node.py): `gello/zmq_core/robot_node.py`
- [Camera ZMQ node](https://github.com/wuphilipp/gello_software/blob/main/gello/zmq_core/camera_node.py): `gello/zmq_core/camera_node.py`

### Simulation and Data

- [Simulation arms](https://github.com/wuphilipp/gello_software/blob/main/gello/dm_control_tasks/arms/manipulator.py): `gello/dm_control_tasks/arms/manipulator.py`
- [Manipulation tasks](https://github.com/wuphilipp/gello_software/tree/main/gello/dm_control_tasks/manipulation/tasks): `gello/dm_control_tasks/manipulation/tasks`
- [Simulation arenas](https://github.com/wuphilipp/gello_software/tree/main/gello/dm_control_tasks/manipulation/arenas): `gello/dm_control_tasks/manipulation/arenas`
- [Mujoco robot](https://github.com/wuphilipp/gello_software/blob/main/gello/robots/sim_robot.py): `gello/robots/sim_robot.py`
- [Demo conversion](https://github.com/wuphilipp/gello_software/blob/main/gello/data_utils/demo_to_gdict.py): `gello/data_utils/demo_to_gdict.py`
- [Conversion utilities](https://github.com/wuphilipp/gello_software/blob/main/gello/data_utils/conversion_utils.py): `gello/data_utils/conversion_utils.py`
- [MJCF utilities](https://github.com/wuphilipp/gello_software/blob/main/gello/dm_control_tasks/mjcf_utils.py): `gello/dm_control_tasks/mjcf_utils.py`

### FACTR Compensation

- [FACTR gravity compensation](https://github.com/wuphilipp/gello_software/blob/main/gello/factr/gravity_compensation.py): `gello/factr/gravity_compensation.py`
- [FACTR YAM agent](https://github.com/wuphilipp/gello_software/blob/main/gello/factr/yam_gello_factr_agent.py): `gello/factr/yam_gello_factr_agent.py`

### ROS 2 Franka

- [GELLO state publisher](https://github.com/wuphilipp/gello_software/blob/main/ros2/src/franka_gello_state_publisher/franka_gello_state_publisher/gello_publisher.py): `ros2/src/franka_gello_state_publisher/franka_gello_state_publisher/gello_publisher.py`
- [Franka GELLO hardware](https://github.com/wuphilipp/gello_software/blob/main/ros2/src/franka_gello_state_publisher/franka_gello_state_publisher/gello_hardware.py): `ros2/src/franka_gello_state_publisher/franka_gello_state_publisher/gello_hardware.py`
- [Joint impedance controller](https://github.com/wuphilipp/gello_software/blob/main/ros2/src/franka_fr3_arm_controllers/src/joint_impedance_controller.cpp): `ros2/src/franka_fr3_arm_controllers/src/joint_impedance_controller.cpp`
- [Motion generator](https://github.com/wuphilipp/gello_software/blob/main/ros2/src/franka_fr3_arm_controllers/src/motion_generator.cpp): `ros2/src/franka_fr3_arm_controllers/src/motion_generator.cpp`
- [Franka gripper client](https://github.com/wuphilipp/gello_software/blob/main/ros2/src/franka_gripper_manager/franka_gripper_manager/franka_gripper_client.py): `ros2/src/franka_gripper_manager/franka_gripper_manager/franka_gripper_client.py`
- [Robotiq gripper client](https://github.com/wuphilipp/gello_software/blob/main/ros2/src/franka_gripper_manager/franka_gripper_manager/robotiq_gripper_client.py): `ros2/src/franka_gripper_manager/franka_gripper_manager/robotiq_gripper_client.py`

### Other components

- **Operator**
- **Robot arm**
- **Camera device**
- **Franka FR3**

## Connections

- Operator → Launch orchestration: starts (evidence: `README.md`)
- Launch orchestration → Control loop: runs (evidence: `gello/utils/launch_utils.py`)
- Control loop → Agent interface: requests action (evidence: `gello/utils/control_utils.py`)
- GELLO agent → Agent interface: implements (evidence: `gello/agents/agent.py`)
- SpaceMouse agent → Agent interface: implements (evidence: `gello/agents/spacemouse_agent.py`)
- Quest agent → Agent interface: implements (evidence: `gello/agents/quest_agent.py`)
- Control loop → Robot environment: steps (evidence: `gello/utils/control_utils.py`)
- Robot environment → Robot interface: commands and reads (evidence: `gello/env.py`)
- Robot environment → Camera interface: reads frames (evidence: `gello/env.py`)
- RealSense camera → Camera interface: implements (evidence: `gello/cameras/realsense_camera.py`)
- Robot drivers → Robot interface: implement (evidence: `gello/robots/xarm_robot.py`)
- Robot drivers → Dynamixel driver: uses (evidence: `gello/robots/dynamixel.py`)
- Robot ZMQ server/client → Robot interface: dispatches methods (evidence: `gello/zmq_core/robot_node.py`)
- Camera ZMQ node → Camera interface: uses (evidence: `gello/zmq_core/camera_node.py`)
- Launch orchestration → Robot ZMQ server/client: configures (evidence: `gello/utils/launch_utils.py`)
- Launch orchestration → Dynamixel driver: initializes (evidence: `gello/utils/launch_utils.py`)
- Robot environment → Robot arm: commands via adapter
- Camera interface → Camera device: reads from
- Manipulation tasks → Simulation arms: uses (evidence: `gello/dm_control_tasks/manipulation/tasks/block_play.py`)
- Manipulation tasks → Simulation arenas: uses (evidence: `gello/dm_control_tasks/manipulation/tasks/block_play.py`)
- Simulation arms → MJCF utilities: uses MJCF helpers (evidence: `gello/dm_control_tasks/arms/manipulator.py`)
- Demo conversion → Conversion utilities: uses (evidence: `gello/data_utils/demo_to_gdict.py`)
- FACTR gravity compensation → Dynamixel driver: uses (evidence: `gello/factr/gravity_compensation.py`)
- FACTR gravity compensation → Robot ZMQ server/client: optionally connects (evidence: `gello/factr/gravity_compensation.py`)
- FACTR gravity compensation → Robot environment: optionally creates (evidence: `gello/factr/gravity_compensation.py`)
- FACTR YAM agent → FACTR gravity compensation: uses (evidence: `gello/factr/yam_gello_factr_agent.py`)
- GELLO state publisher → Franka GELLO hardware: uses hardware interface
- Franka GELLO hardware → Franka FR3: interfaces with
- Joint impedance controller → Motion generator: uses
- Franka gripper client → Franka FR3: controls gripper
- Robotiq gripper client → Franka FR3: controls gripper
