# rail-berkeley/hil-serl architecture

> Architecture diagram of the GitHub repository rail-berkeley/hil-serl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rail-berkeley/hil-serl
- Repository: https://github.com/rail-berkeley/hil-serl

## Overview

##### Main Components
- Actor & Learner Nodes:
- The core idea is an asynchronous architecture where an “actor” interacts with the gym environment (whether simulated or real) and a separate “learner” node receives data and synchronizes policies.
- serl_launcher Module:
- Contains the key libraries for training RL policies. This includes the agents (e.g., SAC, BC policies), network definitions, data management (replay buffers, datasets), and utility functions.
- Break these out into submodules such as “Agents,” “Data,” “Networks,” “Wrappers,” “Vision,” and “Common Utilities.”
- serl_robot_infra Module:
- Responsible for interfacing with real robotic hardware.
- Contains subcomponents like the robot servers, franka_env (gym environment for the robot), specialized controllers (egg_flip_controller), and camera/spacemouse components.
- Examples:
- Houses example scripts for training, data recording, and running various experiments (egg flip, ram insertion, etc.). These illustrate how the above components are integrated.

##### Relationships and Interactions
- Data Flow and Synchronization:
- The actor node collects sensor data and executes the policy, then sends experience and demonstration data to the learner.
- The learner node computes policy updates and periodically synchronizes with the actor—this asynchronous communication happens, for example, via the agentlace library.
- Modular Interactions:
- The “serl_launcher” components (agents, networks, replay buffers) interact closely to perform RL training.
- The “serl_robot_infra” components interface with real robotic systems, issuing commands through ROS (as seen in the robot_servers using Flask servers and ROS launch scripts).
- The examples module then demonstrates integration, by using these libraries and components to run experiments.

##### Architectural Patterns and Key Technologies
- Asynchronous Actor-Learner Pattern:
- That the system uses two nodes operating in parallel, exchanging data over the network.
- Component-Based Design:
- Clearly segregate policy training (agents, networks), environment interaction (wrappers, gym environments), and hardware interfacing (robot servers, controllers).
- Use of External Libraries/Frameworks:
- Jax is used for computation (with support for CPU, GPU, or TPU), ensuring fast training and inference.
- The agentlace library is used for network communication between nodes.
- ROS integration is present in the serl_robot_infra module for real-time robot control.

##### Putting It All Together
- Under “RL Training Environment,” show the serl_launcher package broken into its submodules (Agents, Data, Networks, Wrappers, Vision, etc.) along with the examples illustrating how these are invoked.
- Under “Real Robot Infrastructure,” include robot servers, the egg_flip_controller, franka_env (with cameras and sensor setups), and any integration points with ROS.
- Data flow from sensor inputs (camera, spacemouse) to the gym environments.
- Policy updates flowing from learner to actor.
- Command/control signals from the robot server to the real robot hardware.
- Use annotations to indicate the external libraries (Jax, agentlace, ROS) that facilitate these connections.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% RL Training Environment Section
    subgraph "RL Training Environment"
        Actor["Actor Node"]:::actor
        Learner["Learner Node"]:::learner

        %% serl_launcher Module Subgraph
        subgraph "serl_launcher Module"
            Launcher["Core Library"]:::rl
            Agents["Agents (SAC, BC)"]:::rl
            DataManagement["Data Management"]:::rl
            Networks["Network Definitions"]:::rl
            Wrappers["Environment Wrappers"]:::rl
            Vision["Vision & Augmentations"]:::rl
            CommonUtil["Common Utilities"]:::rl
            Utilities["Utilities"]:::rl
        end

        %% Examples Subgraph
        subgraph "Examples"
            EggFlipActor["Run Actor (Egg Flip)"]:::example
            EggFlipLearner["Run Learner (Egg Flip)"]:::example
        end
    end

    %% Real Robot Infrastructure Section
    subgraph "Real Robot Infrastructure"
        RobotServers["Robot Servers"]:::robot
        FrankaEnv["Franka Environment (Cameras, Spacemouse)"]:::robot
        EggFlipController["Egg Flip Controller"]:::robot
    end

    %% Data Flow between Actor and Learner
    Actor -->|"experience"| Learner
    Learner -->|"policy_updates"| Actor

    %% Interaction between RL Nodes and serl_launcher Module
    Actor -->|"uses"| Launcher
    Learner -->|"updates"| Launcher

    %% Internal connections within serl_launcher Module
    Launcher -->|"integrates"| Agents
    Launcher -->|"manages"| DataManagement
    Launcher -->|"defines"| Networks
    Launcher -->|"wraps"| Wrappers
    Launcher -->|"processes"| Vision
    Launcher -->|"assists"| CommonUtil
    Launcher -->|"utilizes"| Utilities

    %% Examples triggering Actor and Learner nodes
    EggFlipActor -->|"triggers"| Actor
    EggFlipLearner -->|"triggers"| Learner

    %% Interaction between RL Environment and Real Robot Infrastructure
    Actor -->|"sends_control"| FrankaEnv
    FrankaEnv -->|"feedback"| Actor
    Learner -->|"policy_dispatch"| RobotServers
    RobotServers -->|"controls"| EggFlipController

    %% Styling Classes
    classDef actor fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef learner fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef rl fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef example fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef robot fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;

    %% Click Events for Actor & Learner Experiments
    click EggFlipActor "https://github.com/rail-berkeley/hil-serl/blob/main/examples/experiments/egg_flip/run_actor.sh"
    click EggFlipLearner "https://github.com/rail-berkeley/hil-serl/blob/main/examples/experiments/egg_flip/run_learner.sh"

    %% Click Events for serl_launcher Module Components
    click Launcher "https://github.com/rail-berkeley/hil-serl/tree/main/serl_launcher/serl_launcher"
    click Agents "https://github.com/rail-berkeley/hil-serl/tree/main/serl_launcher/serl_launcher/agents/continuous"
    click DataManagement "https://github.com/rail-berkeley/hil-serl/tree/main/serl_launcher/serl_launcher/data"
    click Networks "https://github.com/rail-berkeley/hil-serl/tree/main/serl_launcher/serl_launcher/networks"
    click Wrappers "https://github.com/rail-berkeley/hil-serl/tree/main/serl_launcher/serl_launcher/wrappers"
    click Vision "https://github.com/rail-berkeley/hil-serl/tree/main/serl_launcher/serl_launcher/vision"
    click CommonUtil "https://github.com/rail-berkeley/hil-serl/tree/main/serl_launcher/serl_launcher/common"
    click Utilities "https://github.com/rail-berkeley/hil-serl/tree/main/serl_launcher/serl_launcher/utils"

    %% Click Events for serl_robot_infra Module Components
    click RobotServers "https://github.com/rail-berkeley/hil-serl/tree/main/serl_robot_infra/robot_servers"
    click FrankaEnv "https://github.com/rail-berkeley/hil-serl/tree/main/serl_robot_infra/franka_env"
    click EggFlipController "https://github.com/rail-berkeley/hil-serl/tree/main/serl_robot_infra/egg_flip_controller"
```
