How shawnhymel/reinforcement-learning-for-robotics works
This repository is a robotics reinforcement-learning demonstration, taking a two-wheel balance bot from simulation environments through PPO training and toward embedded inference and remote operation. Its main progression includes baseline, domain-randomized, and command-conditioned simulation, followed by policy deployment on the robot. Separate hardware examples cover sensor checks and IMU calibration. The tree and README establish these product areas; sampled excerpts confirm environment behavior, PPO implementation, embedded actor inference, and WebSocket command input, but do not establish every notebook-to-module connection.
Main entry points: ppo_trainer.py PPO trainer · bala.cpp Bala hardware interface · actor.h Embedded actor · onnx_actor_to_c.py ONNX actor converter
Components 23
Robot model
- CAD and robot assets
workspace/mechanical/FreeCAD/bala2-fire - URDF description
workspace/mechanical/FreeCAD/bala2-fire/bala2-fire-simplified.urdf
Simulation environments
- Balance environment
workspace/software/02-balance-bot-with-ppo/balance_bot_env.py - Baseline PPO demos
workspace/software/02-balance-bot-with-ppo/train_with_ppo.ipynb - Randomized environment
workspace/software/04-domain-randomization/balance_bot_env_dr.py - Randomization training
workspace/software/04-domain-randomization/train_with_ppo_dr.ipynb - Command environment
workspace/software/05-command-conditioning/balance_bot_env_cmd.py - Command training
workspace/software/05-command-conditioning/train_with_ppo_cmd.ipynb - Simulation demos
workspace/software/01-test-model-in-mujoco
Policy learning
- PPO trainer
workspace/software/rl/ppo_trainer.py - ONNX actor converter
workspace/software/rl/onnx_actor_to_c.py
Robot deployment
- Inference demo
workspace/software/03-sim-to-real/inference_test.ipynb - Robot firmware
workspace/software/03-sim-to-real/balance_bot/balance_bot.ino - Embedded actor
workspace/software/03-sim-to-real/balance_bot/actor.h - Static inference demo
workspace/software/03-sim-to-real/static_inference_test/static_inference_test.ino - Randomized-policy firmware
workspace/software/04-domain-randomization/balance_bot_fixes/balance_bot_fixes.ino - Remote-control firmware
workspace/software/06-remote-control/balance_bot_rc/balance_bot_rc.ino - Thumbstick interface
workspace/software/06-remote-control/balance_bot_rc/index_html.h
Hardware controls
- Bala hardware interface
workspace/software/03-sim-to-real/sensor_test/bala.cpp - Sensor test firmware
workspace/software/03-sim-to-real/sensor_test/sensor_test.ino - IMU calibration
workspace/software/03-sim-to-real/imu_calibration/imu_calibration.ino
Other components
- Learner
- Remote operator
Connections 24
- Learner → Simulation demos: explores
- URDF description → CAD and robot assets: describes robot
- Baseline PPO demos → Balance environment: trains in
- Randomization training → Randomized environment: trains in
- Command training → Command environment: trains in
- Baseline PPO demos → PPO trainer: uses trainer
- Randomization training → PPO trainer: uses trainer
- Command training → PPO trainer: uses trainer
- Balance environment → PPO trainer: provides episodes
- Randomized environment → PPO trainer: provides episodes
- Command environment → PPO trainer: provides episodes
- Inference demo → Embedded actor: tests inference
- Robot firmware → Embedded actor: runs actor
- Robot firmware → Bala hardware interface: controls hardware
- Sensor test firmware → Bala hardware interface: uses interface
- Static inference demo → Embedded actor: tests inference
- Randomized-policy firmware → Bala hardware interface: controls hardware
- Remote-control firmware → Bala hardware interface: controls hardware
- Remote operator → Thumbstick interface: operates
- Thumbstick interface → Remote-control firmware: sends commands
- IMU calibration → Bala hardware interface: uses interface
- ONNX actor converter → Embedded actor: generates actor
- ONNX actor converter → Randomized-policy firmware: generates actor
- ONNX actor converter → Remote-control firmware: generates actor