# ubtech-robot/globalhumanoidrobotchallenge_2026_baseline architecture

> Architecture diagram of the GitHub repository ubtech-robot/globalhumanoidrobotchallenge_2026_baseline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline
- Repository: https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline

## Overview

1. **Overall shape**
   - This repository is a robotics baseline built on the LeRobot framework, with a custom **Ubtech_sim** Isaac Sim package plus an extended **src/lerobot** stack for teleoperation, data collection, training, and policy deployment.
   - The architecture is pipeline-oriented: **inputs from cameras/teleoperators/robots → processors → datasets/training → policies → runtime inference/control**.

2. **Simulation layer: `Ubtech_sim/`**
   - `Ubtech_sim/main.py` is the entry into a standalone Isaac Sim simulation workflow.
   - `SceneBuilder`, `RobotArticulation`, `DualArmIK`, `grasp_planner`, and `coordinate_utils` together define the simulated world, robot kinematics, grasping logic, and coordinate transforms.
   - YAML configs such as `Conveyor_Sorting.yaml`, `Foam_Inlaying.yaml`, `Packing_Box.yaml`, and `Part_Sorting.yaml` indicate task-scene specialization; `DataLogger` suggests simulation-side logging for later dataset creation or debugging.

3. **Framework boundary: `src/lerobot/`**
   - `src/lerobot` is the reusable robotics framework layer. It is not just one application; it contains abstractions for robots, cameras, teleoperators, datasets, policies, RL, and deployment.
   - The GHRC baseline uses this framework to standardize how observations, actions, and models are connected across simulation and real/virtual hardware.

4. **Robot and actuation abstraction**
   - `src/lerobot/robots/` defines follower robot interfaces and robot-specific adapters for many platforms; the competition-relevant simulation path is centered on **Walker S2** support under `walker_s2_sim`.
   - `src/lerobot/motors/` is the low-level hardware boundary, with separate implementations for actuator families such as Dynamixel, Feetech, Damiao, and Robstride.
   - This layer isolates model-level control from the underlying robot hardware or simulator implementation.

5. **Teleoperation boundary**
   - `src/lerobot/teleoperators/` contains leader-device adapters: keyboard, gamepad, phone, leader arms, gloves, and robot-specific teleoperators.
   - For the GHRC baseline, the key interaction mode is **keyboard teleoperation** (`walker_s2_keyboard` and the generic keyboard teleop modules), which turns human input into normalized robot actions.
   - This layer sits upstream of data recording and live control, acting as the human-to-robot input source.

6. **Camera and sensor subsystem**
   - `src/lerobot/cameras/` provides a camera abstraction with backends for OpenCV, RealSense, Reachy2, and ZMQ.
   - The 4-camera real-time display described in the README is naturally modeled here: camera streams are captured, optionally published over ZMQ, and consumed by teleop, replay, recording, or inference UIs.
   - `image_server.py` and the ZMQ camera backend form the key transport boundary for remote visualization.

7. **Data processing and dataset flow**
   - `src/lerobot/processor/` is the normalization/conversion layer between raw robot/camera signals and model-ready tensors.
   - It includes processors for observations, actions, tokenization, renaming, slicing, normalization, batching, and robot-policy bridging.
   - `src/lerobot/datasets/` stores the dataset stack, including the LeRobot dataset implementation, streaming support, image/video utilities, statistics, transforms, and dataset publishing helpers.
   - The README’s dataset format requirement maps to this subsystem: recording produces standardized **LeRobotDataset V3.0** artifacts, with Hugging Face used for external dataset storage/distribution.

8. **Training and policy families**
   - `src/lerobot/policies/` contains multiple model families: ACT, Diffusion, Pi0/Pi0.5, SmolVLA, VQBeT, SARM, TDMPC, RTC, GROOT, XVLA, and others.
   - Each policy typically has its own configuration, model, and processor, which means training is structured as: **dataset batch → policy-specific processor → model forward/backward → optimizer/scheduler**.
   - `src/lerobot/optim/` and `src/lerobot/configs/` provide the training-time control plane for hyperparameters, optimizer selection, and run configuration.

9. **Inference and online execution**
   - `src/lerobot/async_inference/` supports a split execution model with a **policy server** and **robot client**, enabling policies to run remotely or asynchronously from the robot loop.
   - `src/lerobot/transport/` contains gRPC service definitions and generated stubs, which are the explicit network boundary for this inference/control split.
   - This is the main path for low-latency deployment: observations are sent to the policy process, actions are returned, and the robot client executes them while optionally recording outputs.

10. **RL and evaluation tooling**
   - `src/lerobot/rl/` adds a reinforcement-learning-oriented loop with actor/learner components, buffers, queues, evaluation helpers, and WandB integration.
   - `src/lerobot/envs/` wraps external benchmark environments such as LIBERO and MetaWorld, showing that the framework supports more than just the GHRC humanoid simulation path.
   - These modules form a separate training/eval loop from imitation learning, but share the same dataset, processor, and policy infrastructure.

11. **Competition-specific Walker S2 path**
   - The README’s main baseline is the **Walker S2 simulation + keyboard teleoperation + 4-camera view + imitation learning** workflow.
   - In code terms, that path is implemented by the Walker S2 simulator modules under `robots/walker_s2_sim`, the keyboard teleoperator modules, the camera stack, and the LeRobot dataset/policy subsystems.
   - This is the core architecture to draw if focusing on the GHRC 2026 baseline.

12. **Entry points and operational tooling**
   - `src/lerobot/scripts/` contains the runnable entry scripts for the major lifecycle steps: setup, calibration, camera discovery, teleoperation, recording, replay, training, evaluation, and dataset editing.
   - `Dockerfile`, `run.sh`, `pyproject.toml`, and `setup.py` indicate a containerized, package-managed runtime intended to make Isaac Sim, data collection, and training reproducible.
   - The repo also relies on external assets via the `assets/` submodule and Hugging Face-hosted resources for large simulation and dataset files.

Main entry points: [`src/lerobot/scripts/lerobot_record.py`](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/scripts/lerobot_record.py) (Lifecycle scripts), [`src/lerobot/teleoperators/walker_s2_keyboard/teleop.py`](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/teleoperators/walker_s2_keyboard/teleop.py) (Keyboard teleop), [`Ubtech_sim/main.py`](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/Ubtech_sim/main.py) (Simulation entry), [`src/lerobot/policies/factory.py`](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/policies/factory.py) (Policy registry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sim["Ubtech Sim"]
  node_ubtech_main["Simulation entry<br/>isaac-sim entry<br/>[main.py]"]
  node_scene_builder["Scene builder<br/>scene assembly<br/>[SceneBuilder.py]"]
  node_robot_articulation["Robot articulation<br/>robot kinematics"]
  node_dual_arm_ik["Dual-arm IK<br/>inverse kinematics<br/>[DualArmIK.py]"]
  node_grasp_planner["Grasp planner<br/>grasping logic<br/>[grasp_planner.py]"]
  node_data_logger["Data logger<br/>simulation logging<br/>[DataLogger.py]"]
end

subgraph group_framework["LeRobot Stack"]
  node_walker_sim["Walker sim<br/>robot interface<br/>[walkers2sim.py]"]
  node_keyboard_teleop["Keyboard teleop<br/>human input<br/>[teleop.py]"]
  node_camera_core["Camera stack<br/>sensor abstraction<br/>[camera.py]"]
  node_image_server["Image server<br/>video transport<br/>[image_server.py]"]
  node_processor_pipe["Processor pipe<br/>signal processing<br/>[pipeline.py]"]
  node_dataset_core["Dataset core<br/>dataset stack<br/>[lerobot_dataset.py]"]
  node_policy_registry["Policy registry<br/>model zoo<br/>[factory.py]"]
  node_policy_act["ACT policy<br/>imitation policy<br/>[modeling_act.py]"]
  node_policy_pi0["Pi0 policy<br/>vision-language policy<br/>[modeling_pi0.py]"]
  node_train_cfg["Train config<br/>training config<br/>[train.py]"]
  node_optim_factory["Optim factory<br/>optimization<br/>[factory.py]"]
  node_async_server["Policy server<br/>async inference<br/>[policy_server.py]"]
  node_grpc_transport["gRPC transport<br/>network boundary<br/>[services.proto]"]
end

subgraph group_ops["Lifecycle Tools"]
  node_record_train_tools["Lifecycle scripts<br/>cli entrypoints<br/>[lerobot_record.py]"]
end

node_ubtech_main -->|"builds"| node_scene_builder
node_ubtech_main -->|"instantiates"| node_robot_articulation
node_scene_builder -->|"uses"| node_grasp_planner
node_robot_articulation -->|"drives"| node_dual_arm_ik
node_ubtech_main -->|"logs"| node_data_logger
node_keyboard_teleop -->|"commands"| node_walker_sim
node_camera_core -->|"streams"| node_image_server
node_image_server -.->|"supports"| node_keyboard_teleop
node_walker_sim -->|"feeds"| node_processor_pipe
node_keyboard_teleop -->|"inputs"| node_processor_pipe
node_processor_pipe -->|"writes"| node_dataset_core
node_dataset_core -->|"trains"| node_policy_registry
node_policy_registry -->|"selects"| node_policy_act
node_policy_registry -->|"selects"| node_policy_pi0
node_train_cfg -->|"configures"| node_optim_factory
node_policy_act -->|"optimizes"| node_optim_factory
node_policy_pi0 -->|"optimizes"| node_optim_factory
node_async_server -->|"serves"| node_grpc_transport
node_grpc_transport -->|"controls"| node_walker_sim
node_record_train_tools -->|"launches"| node_keyboard_teleop
node_record_train_tools -->|"records"| node_dataset_core
node_record_train_tools -->|"runs"| node_train_cfg
node_record_train_tools -->|"deploys"| node_async_server

click node_ubtech_main "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/Ubtech_sim/main.py"
click node_scene_builder "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/Ubtech_sim/source/SceneBuilder.py"
click node_robot_articulation "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/Ubtech_sim/source/RobotArticulation.py"
click node_dual_arm_ik "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/Ubtech_sim/source/DualArmIK.py"
click node_grasp_planner "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/Ubtech_sim/source/grasp_planner.py"
click node_data_logger "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/Ubtech_sim/source/DataLogger.py"
click node_walker_sim "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/robots/walker_s2_sim/walkers2sim.py"
click node_keyboard_teleop "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/teleoperators/walker_s2_keyboard/teleop.py"
click node_camera_core "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/cameras/camera.py"
click node_image_server "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/cameras/zmq/image_server.py"
click node_processor_pipe "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/processor/pipeline.py"
click node_dataset_core "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/datasets/lerobot_dataset.py"
click node_policy_registry "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/policies/factory.py"
click node_policy_act "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/policies/act/modeling_act.py"
click node_policy_pi0 "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/policies/pi0/modeling_pi0.py"
click node_train_cfg "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/configs/train.py"
click node_optim_factory "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/optim/factory.py"
click node_async_server "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/async_inference/policy_server.py"
click node_grpc_transport "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/transport/services.proto"
click node_record_train_tools "https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/scripts/lerobot_record.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ubtech_main,node_scene_builder,node_robot_articulation,node_dual_arm_ik,node_grasp_planner,node_data_logger toneBlue
class node_walker_sim,node_keyboard_teleop,node_camera_core,node_image_server,node_processor_pipe,node_dataset_core,node_policy_registry,node_policy_act,node_policy_pi0,node_train_cfg,node_optim_factory,node_async_server,node_grpc_transport toneAmber
class node_record_train_tools toneMint
```

## Components

### Ubtech Sim

Standalone Isaac Sim workflow for task scenes, kinematics, grasping, and logging.

- [Simulation entry](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/Ubtech_sim/main.py): isaac-sim entry · Starts the standalone Ubtech_sim Isaac Sim workflow. · `Ubtech_sim/main.py`
- [Scene builder](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/Ubtech_sim/source/SceneBuilder.py): scene assembly · Builds task-specific simulation worlds and assets. · `Ubtech_sim/source/SceneBuilder.py`
- [Robot articulation](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/Ubtech_sim/source/RobotArticulation.py): robot kinematics · Defines robot body structure and articulation control in simulation. · `Ubtech_sim/source/RobotArticulation.py`
- [Dual-arm IK](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/Ubtech_sim/source/DualArmIK.py): inverse kinematics · Solves arm motion and pose targeting for the simulated robot. · `Ubtech_sim/source/DualArmIK.py`
- [Grasp planner](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/Ubtech_sim/source/grasp_planner.py): grasping logic · Generates task grasp strategies for simulated manipulation. · `Ubtech_sim/source/grasp_planner.py`
- [Data logger](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/Ubtech_sim/source/DataLogger.py): simulation logging · Records simulation outputs for debugging and dataset creation. · `Ubtech_sim/source/DataLogger.py`

### LeRobot Stack

Reusable robotics framework for robots, cameras, teleop, data, training, and deployment.

- [Walker sim](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/robots/walker_s2_sim/walkers2sim.py): robot interface · Isaac Sim-facing Walker S2 simulation adapter and runtime interface. · `src/lerobot/robots/walker_s2_sim/walkers2sim.py`
- [Keyboard teleop](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/teleoperators/walker_s2_keyboard/teleop.py): human input · Maps keyboard input to normalized Walker S2 teleoperation commands. · `src/lerobot/teleoperators/walker_s2_keyboard/teleop.py`
- [Camera stack](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/cameras/camera.py): sensor abstraction · Common camera abstraction shared across OpenCV, RealSense, ZMQ, and device-specific backends. · `src/lerobot/cameras/camera.py`
- [Image server](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/cameras/zmq/image_server.py): video transport · Publishes camera streams for remote viewing and distributed teleop/inference. · `src/lerobot/cameras/zmq/image_server.py`
- [Processor pipe](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/processor/pipeline.py): signal processing · Normalizes, renames, slices, and bridges observations/actions into model-ready data. · `src/lerobot/processor/pipeline.py`
- [Dataset core](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/datasets/lerobot_dataset.py): dataset stack · LeRobot dataset implementation, streaming, transforms, and publishing helpers. · `src/lerobot/datasets/lerobot_dataset.py`
- [Policy registry](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/policies/factory.py): model zoo · Selects and wires policy families with their processors and configs. · `src/lerobot/policies/factory.py`
- [ACT policy](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/policies/act/modeling_act.py): imitation policy · Representative action-sequence policy family used during training and inference. · `src/lerobot/policies/act/modeling_act.py`
- [Pi0 policy](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/policies/pi0/modeling_pi0.py): vision-language policy · Representative multimodal policy family for robot control. · `src/lerobot/policies/pi0/modeling_pi0.py`
- [Train config](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/configs/train.py): training config · Training and evaluation configuration control plane. · `src/lerobot/configs/train.py`
- [Optim factory](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/optim/factory.py): optimization · Creates optimizers and schedulers for policy training. · `src/lerobot/optim/factory.py`
- [Policy server](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/async_inference/policy_server.py): async inference · Runs policies remotely or asynchronously from the robot loop. · `src/lerobot/async_inference/policy_server.py`
- [gRPC transport](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/transport/services.proto): network boundary · Defines the robot-client / policy-server service interface. · `src/lerobot/transport/services.proto`

### Lifecycle Tools

Entry scripts and runtime orchestration for setup, record, replay, train, and eval.

- [Lifecycle scripts](https://github.com/ubtech-robot/globalhumanoidrobotchallenge_2026_baseline/blob/main/src/lerobot/scripts/lerobot_record.py): cli entrypoints · Operational scripts for setup, calibration, teleop, record, replay, train, and eval. · `src/lerobot/scripts/lerobot_record.py`

## Connections

- Simulation entry → Scene builder: builds · Main simulation workflow constructs the world and task scene.
- Simulation entry → Robot articulation: instantiates · Creates the simulated robot articulation.
- Scene builder → Grasp planner: uses · Scene/task setup relies on grasp logic.
- Robot articulation → Dual-arm IK: drives · Robot motion is generated through kinematic solving.
- Simulation entry → Data logger: logs · Simulation records outputs for later reuse.
- Keyboard teleop → Walker sim: commands · Human keyboard input becomes robot actions.
- Camera stack → Image server: streams · Camera frames can be published for remote visualization.
- Image server → Keyboard teleop: supports · Remote view aids teleoperation and control.
- Walker sim → Processor pipe: feeds · Robot observations/actions are normalized before downstream use.
- Keyboard teleop → Processor pipe: inputs · Teleop actions flow through shared processing and normalization.
- Processor pipe → Dataset core: writes · Processed trajectories are stored as datasets.
- Dataset core → Policy registry: trains · Datasets feed policy training and evaluation.
- Policy registry → ACT policy: selects · Policy factory can instantiate ACT-family models.
- Policy registry → Pi0 policy: selects · Policy factory can instantiate Pi0-family models.
- Train config → Optim factory: configures · Training config determines optimization behavior.
- ACT policy → Optim factory: optimizes · Policy training uses shared optimizer/scheduler plumbing.
- Pi0 policy → Optim factory: optimizes · Policy training uses shared optimizer/scheduler plumbing.
- Policy server → gRPC transport: serves · Asynchronous inference is exposed through the service boundary.
- gRPC transport → Walker sim: controls · Robot client path carries observations and actions to the simulator-facing runtime.
- Lifecycle scripts → Keyboard teleop: launches · Scripts start teleoperation and recording workflows.
- Lifecycle scripts → Dataset core: records · Scripts manage dataset creation and editing.
- Lifecycle scripts → Train config: runs · Scripts launch training and evaluation pipelines.
- Lifecycle scripts → Policy server: deploys · Scripts can start inference and evaluation loops.