How rail-berkeley/hil-serl works
Main Components
• Actor & Learner Nodes:
• The core idea is an asynchronous architecture where an “actor” interacts with the gym environment (whether simulated or real) and a separate “learner” node receives data and synchronizes policies.
• serl_launcher Module:
• Contains the key libraries for training RL policies. This includes the agents (e.g., SAC, BC policies), network definitions, data management (replay buffers, datasets), and utility functions.
• Break these out into submodules such as “Agents,” “Data,” “Networks,” “Wrappers,” “Vision,” and “Common Utilities.”
• serl_robot_infra Module:
• Responsible for interfacing with real robotic hardware.
• Contains subcomponents like the robot servers, franka_env (gym environment for the robot), specialized controllers (egg_flip_controller), and camera/spacemouse components.
• Examples:
• Houses example scripts for training, data recording, and running various experiments (egg flip, ram insertion, etc.). These illustrate how the above components are integrated.
Relationships and Interactions
• Data Flow and Synchronization:
• The actor node collects sensor data and executes the policy, then sends experience and demonstration data to the learner.
• The learner node computes policy updates and periodically synchronizes with the actor—this asynchronous communication happens, for example, via the agentlace library.
• Modular Interactions:
• The “serl_launcher” components (agents, networks, replay buffers) interact closely to perform RL training.
• The “serl_robot_infra” components interface with real robotic systems, issuing commands through ROS (as seen in the robot_servers using Flask servers and ROS launch scripts).
• The examples module then demonstrates integration, by using these libraries and components to run experiments.
Architectural Patterns and Key Technologies
• Asynchronous Actor-Learner Pattern:
• That the system uses two nodes operating in parallel, exchanging data over the network.
• Component-Based Design:
• Clearly segregate policy training (agents, networks), environment interaction (wrappers, gym environments), and hardware interfacing (robot servers, controllers).
• Use of External Libraries/Frameworks:
• Jax is used for computation (with support for CPU, GPU, or TPU), ensuring fast training and inference.
• The agentlace library is used for network communication between nodes.
• ROS integration is present in the serl_robot_infra module for real-time robot control.
Putting It All Together
• Under “RL Training Environment,” show the serl_launcher package broken into its submodules (Agents, Data, Networks, Wrappers, Vision, etc.) along with the examples illustrating how these are invoked.
• Under “Real Robot Infrastructure,” include robot servers, the egg_flip_controller, franka_env (with cameras and sensor setups), and any integration points with ROS.
• Data flow from sensor inputs (camera, spacemouse) to the gym environments.
• Policy updates flowing from learner to actor.
• Command/control signals from the robot server to the real robot hardware.
• Use annotations to indicate the external libraries (Jax, agentlace, ROS) that facilitate these connections.