How zecice/so101-pingpong-balance works
This project balances a ping-pong ball on an SO-101 paddle, training a PPO controller in MuJoCo and running that controller on a real arm. The main runtime path is wrist-camera image → orange-blob detection → ball-position estimation → filtered velocity → controller → joint commands; simulation supplies training and evaluation, while hardware setup and safety utilities support deployment. The README documents the end-to-end workflow; source excerpts and the SOURCE INDEX support the internal links shown. Some unsampled implementation details are omitted rather than inferred.
Main entry points: run_real.py Real-arm loop · estimator.py Ball estimator · policies.py Controller policy · env.py Balance environment
Components 21
Simulation and control
- Filtered velocity
sim/env.py - Controller policy
sim/policies.py - PPO actor-critic
sim/ppo.py - Balance environment
sim/env.py - Parallel environments
sim/vec_env.py - RAM guard
sim/ram.py - Controller training
sim/train_controller.py - Policy evaluation
sim/eval_policy.py - Simulation viewer
sim/view_controller.py
Ball perception
- Orange detector
perception/detector.py - Ball estimator
perception/estimator.py
Real-arm operation
- Wrist camera
hardware/camera.py - Real-arm loop
hardware/run_real.py - Home pose setup
hardware/pose_guide.py - Latency measurement
hardware/measure_latency.py - Torque release
hardware/torque_off.py
Models and weights
- Arm and scene model
assets/so101/scene_balance.xml - Controller weights
weights/controller.pt - Estimator weights
weights/estimator.pt
Other components
- SO-101 arm
- Ping-pong ball
Connections 31
- Wrist camera → Orange detector: provides images
README.md - Orange detector → Ball estimator: provides detections
README.md - Ball estimator → Filtered velocity: provides positions
README.md - Filtered velocity → Controller policy: provides state
README.md - Controller policy → Real-arm loop: supplies actions
README.md - Real-arm loop → SO-101 arm: commands joints
hardware/run_real.py - Ping-pong ball → Orange detector: appears in image
README.md - Controller training → Parallel environments: collects rollouts
sim/train_controller.py - Controller training → PPO actor-critic: updates policy
sim/train_controller.py - Parallel environments → Balance environment: steps environments
sim/vec_env.py - Parallel environments → RAM guard: checks memory
sim/vec_env.py - Balance environment → Arm and scene model: uses scene model
- Policy evaluation → Balance environment: evaluates in
sim/eval_policy.py - Policy evaluation → Controller policy: queries policy
sim/eval_policy.py - Simulation viewer → Balance environment: steps environment
sim/view_controller.py - Simulation viewer → Controller policy: queries policy
sim/view_controller.py - Real-arm loop → Wrist camera: opens camera
hardware/run_real.py - Real-arm loop → Orange detector: detects ball
hardware/run_real.py - Real-arm loop → Ball estimator: estimates position
hardware/run_real.py - Real-arm loop → Filtered velocity: filters velocity
README.md - Real-arm loop → Controller weights: loads weights
README.md - Real-arm loop → Estimator weights: loads weights
README.md - Controller training → Controller weights: produces checkpoint
README.md - Ball estimator → Estimator weights: trains weights
README.md - Ball estimator → Balance environment: collects simulated data
perception/estimator.py - Ball estimator → Orange detector: detects training frames
perception/estimator.py - Ball estimator → Controller policy: uses balancing policy
perception/estimator.py - Home pose setup → SO-101 arm: guides pose setup
hardware/pose_guide.py - Real-arm loop → Home pose setup: uses saved home pose
README.md - Latency measurement → Wrist camera: measures camera delay
README.md - Torque release → SO-101 arm: releases torque
README.md