# shawnhymel/reinforcement-learning-for-robotics architecture

> Architecture diagram of the GitHub repository shawnhymel/reinforcement-learning-for-robotics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shawnhymel/reinforcement-learning-for-robotics
- Repository: https://github.com/shawnhymel/reinforcement-learning-for-robotics

## Overview

This repository is a robotics reinforcement-learning demonstration, taking a two-wheel balance bot from simulation environments through PPO training and toward embedded inference and remote operation. Its main progression includes baseline, domain-randomized, and command-conditioned simulation, followed by policy deployment on the robot. Separate hardware examples cover sensor checks and IMU calibration. The tree and README establish these product areas; sampled excerpts confirm environment behavior, PPO implementation, embedded actor inference, and WebSocket command input, but do not establish every notebook-to-module connection.

Main entry points: [`workspace/software/rl/ppo_trainer.py`](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/rl/ppo_trainer.py) (PPO trainer), [`workspace/software/03-sim-to-real/sensor_test/bala.cpp`](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/sensor_test/bala.cpp) (Bala hardware interface), [`workspace/software/03-sim-to-real/balance_bot/actor.h`](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/balance_bot/actor.h) (Embedded actor), [`workspace/software/rl/onnx_actor_to_c.py`](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/rl/onnx_actor_to_c.py) (ONNX actor converter)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_model["Robot model"]
  node_cad["CAD and robot assets"]
  node_robot_description["URDF description"]
end

subgraph group_simulation["Simulation environments"]
  node_baseline_env["Balance environment<br/>[balance_bot_env.py]"]
  node_baseline_training["Baseline PPO demos"]
  node_dr_env["Randomized environment"]
  node_dr_training["Randomization training"]
  node_cmd_env["Command environment"]
  node_cmd_training["Command training"]
  node_mujoco_tests["Simulation demos"]
end

subgraph group_learning["Policy learning"]
  node_ppo["PPO trainer<br/>[ppo_trainer.py]"]
  node_export["ONNX actor converter<br/>[onnx_actor_to_c.py]"]
end

subgraph group_deployment["Robot deployment"]
  node_inference_test["Inference demo"]
  node_robot_firmware["Robot firmware<br/>[balance_bot.ino]"]
  node_robot_actor["Embedded actor<br/>[actor.h]"]
  node_static_test["Static inference demo"]
  node_dr_firmware["Randomized-policy firmware"]
  node_command_firmware["Remote-control firmware<br/>[balance_bot_rc.ino]"]
  node_remote_page["Thumbstick interface<br/>[index_html.h]"]
end

subgraph group_controls["Hardware controls"]
  node_hardware_lib["Bala hardware interface<br/>[bala.cpp]"]
  node_sensor_test["Sensor test firmware<br/>[sensor_test.ino]"]
  node_imu_calibration["IMU calibration"]
end

node_learner(("Learner"))
node_browser(("Remote operator"))

node_learner -->|"explores"| node_mujoco_tests
node_robot_description -->|"describes robot"| node_cad
node_baseline_training -->|"trains in"| node_baseline_env
node_dr_training -->|"trains in"| node_dr_env
node_cmd_training -->|"trains in"| node_cmd_env
node_baseline_training -->|"uses trainer"| node_ppo
node_dr_training -->|"uses trainer"| node_ppo
node_cmd_training -->|"uses trainer"| node_ppo
node_baseline_env -->|"provides episodes"| node_ppo
node_dr_env -->|"provides episodes"| node_ppo
node_cmd_env -->|"provides episodes"| node_ppo
node_inference_test -->|"tests inference"| node_robot_actor
node_robot_firmware -->|"runs actor"| node_robot_actor
node_robot_firmware -->|"controls hardware"| node_hardware_lib
node_sensor_test -->|"uses interface"| node_hardware_lib
node_static_test -->|"tests inference"| node_robot_actor
node_dr_firmware -->|"controls hardware"| node_hardware_lib
node_command_firmware -->|"controls hardware"| node_hardware_lib
node_browser -->|"operates"| node_remote_page
node_remote_page -->|"sends commands"| node_command_firmware
node_imu_calibration -->|"uses interface"| node_hardware_lib
node_export -->|"generates actor"| node_robot_actor
node_export -.->|"generates actor"| node_dr_firmware
node_export -.->|"generates actor"| node_command_firmware

click node_cad "https://github.com/shawnhymel/reinforcement-learning-for-robotics/tree/main/workspace/mechanical/FreeCAD/bala2-fire"
click node_robot_description "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/mechanical/FreeCAD/bala2-fire/bala2-fire-simplified.urdf"
click node_baseline_env "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/02-balance-bot-with-ppo/balance_bot_env.py"
click node_baseline_training "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/02-balance-bot-with-ppo/train_with_ppo.ipynb"
click node_dr_env "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/04-domain-randomization/balance_bot_env_dr.py"
click node_dr_training "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/04-domain-randomization/train_with_ppo_dr.ipynb"
click node_cmd_env "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/05-command-conditioning/balance_bot_env_cmd.py"
click node_cmd_training "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/05-command-conditioning/train_with_ppo_cmd.ipynb"
click node_mujoco_tests "https://github.com/shawnhymel/reinforcement-learning-for-robotics/tree/main/workspace/software/01-test-model-in-mujoco"
click node_ppo "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/rl/ppo_trainer.py"
click node_export "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/rl/onnx_actor_to_c.py"
click node_inference_test "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/inference_test.ipynb"
click node_robot_firmware "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/balance_bot/balance_bot.ino"
click node_robot_actor "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/balance_bot/actor.h"
click node_static_test "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/static_inference_test/static_inference_test.ino"
click node_dr_firmware "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/04-domain-randomization/balance_bot_fixes/balance_bot_fixes.ino"
click node_command_firmware "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/06-remote-control/balance_bot_rc/balance_bot_rc.ino"
click node_remote_page "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/06-remote-control/balance_bot_rc/index_html.h"
click node_hardware_lib "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/sensor_test/bala.cpp"
click node_sensor_test "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/sensor_test/sensor_test.ino"
click node_imu_calibration "https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/imu_calibration/imu_calibration.ino"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cad,node_robot_description toneBlue
class node_baseline_env,node_baseline_training,node_dr_env,node_dr_training,node_cmd_env,node_cmd_training,node_mujoco_tests toneAmber
class node_ppo,node_export toneMint
class node_inference_test,node_robot_firmware,node_robot_actor,node_static_test,node_dr_firmware,node_command_firmware,node_remote_page toneRose
class node_hardware_lib,node_sensor_test,node_imu_calibration,node_learner,node_browser toneIndigo
```

## Components

### Robot model

- [CAD and robot assets](https://github.com/shawnhymel/reinforcement-learning-for-robotics/tree/main/workspace/mechanical/FreeCAD/bala2-fire): `workspace/mechanical/FreeCAD/bala2-fire`
- [URDF description](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/mechanical/FreeCAD/bala2-fire/bala2-fire-simplified.urdf): `workspace/mechanical/FreeCAD/bala2-fire/bala2-fire-simplified.urdf`

### Simulation environments

- [Balance environment](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/02-balance-bot-with-ppo/balance_bot_env.py): `workspace/software/02-balance-bot-with-ppo/balance_bot_env.py`
- [Baseline PPO demos](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/02-balance-bot-with-ppo/train_with_ppo.ipynb): `workspace/software/02-balance-bot-with-ppo/train_with_ppo.ipynb`
- [Randomized environment](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/04-domain-randomization/balance_bot_env_dr.py): `workspace/software/04-domain-randomization/balance_bot_env_dr.py`
- [Randomization training](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/04-domain-randomization/train_with_ppo_dr.ipynb): `workspace/software/04-domain-randomization/train_with_ppo_dr.ipynb`
- [Command environment](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/05-command-conditioning/balance_bot_env_cmd.py): `workspace/software/05-command-conditioning/balance_bot_env_cmd.py`
- [Command training](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/05-command-conditioning/train_with_ppo_cmd.ipynb): `workspace/software/05-command-conditioning/train_with_ppo_cmd.ipynb`
- [Simulation demos](https://github.com/shawnhymel/reinforcement-learning-for-robotics/tree/main/workspace/software/01-test-model-in-mujoco): `workspace/software/01-test-model-in-mujoco`

### Policy learning

- [PPO trainer](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/rl/ppo_trainer.py): `workspace/software/rl/ppo_trainer.py`
- [ONNX actor converter](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/rl/onnx_actor_to_c.py): `workspace/software/rl/onnx_actor_to_c.py`

### Robot deployment

- [Inference demo](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/inference_test.ipynb): `workspace/software/03-sim-to-real/inference_test.ipynb`
- [Robot firmware](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/balance_bot/balance_bot.ino): `workspace/software/03-sim-to-real/balance_bot/balance_bot.ino`
- [Embedded actor](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/balance_bot/actor.h): `workspace/software/03-sim-to-real/balance_bot/actor.h`
- [Static inference demo](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/static_inference_test/static_inference_test.ino): `workspace/software/03-sim-to-real/static_inference_test/static_inference_test.ino`
- [Randomized-policy firmware](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/04-domain-randomization/balance_bot_fixes/balance_bot_fixes.ino): `workspace/software/04-domain-randomization/balance_bot_fixes/balance_bot_fixes.ino`
- [Remote-control firmware](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/06-remote-control/balance_bot_rc/balance_bot_rc.ino): `workspace/software/06-remote-control/balance_bot_rc/balance_bot_rc.ino`
- [Thumbstick interface](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/06-remote-control/balance_bot_rc/index_html.h): `workspace/software/06-remote-control/balance_bot_rc/index_html.h`

### Hardware controls

- [Bala hardware interface](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/sensor_test/bala.cpp): `workspace/software/03-sim-to-real/sensor_test/bala.cpp`
- [Sensor test firmware](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/sensor_test/sensor_test.ino): `workspace/software/03-sim-to-real/sensor_test/sensor_test.ino`
- [IMU calibration](https://github.com/shawnhymel/reinforcement-learning-for-robotics/blob/main/workspace/software/03-sim-to-real/imu_calibration/imu_calibration.ino): `workspace/software/03-sim-to-real/imu_calibration/imu_calibration.ino`

### Other components

- **Learner**
- **Remote operator**

## Connections

- Learner → Simulation demos: explores
- URDF description → CAD and robot assets: describes robot
- Baseline PPO demos → Balance environment: trains in
- Randomization training → Randomized environment: trains in
- Command training → Command environment: trains in
- Baseline PPO demos → PPO trainer: uses trainer
- Randomization training → PPO trainer: uses trainer
- Command training → PPO trainer: uses trainer
- Balance environment → PPO trainer: provides episodes
- Randomized environment → PPO trainer: provides episodes
- Command environment → PPO trainer: provides episodes
- Inference demo → Embedded actor: tests inference
- Robot firmware → Embedded actor: runs actor
- Robot firmware → Bala hardware interface: controls hardware
- Sensor test firmware → Bala hardware interface: uses interface
- Static inference demo → Embedded actor: tests inference
- Randomized-policy firmware → Bala hardware interface: controls hardware
- Remote-control firmware → Bala hardware interface: controls hardware
- Remote operator → Thumbstick interface: operates
- Thumbstick interface → Remote-control firmware: sends commands
- IMU calibration → Bala hardware interface: uses interface
- ONNX actor converter → Embedded actor: generates actor
- ONNX actor converter → Randomized-policy firmware: generates actor
- ONNX actor converter → Remote-control firmware: generates actor