# leggedrobotics/rsl_rl architecture

> Architecture diagram of the GitHub repository leggedrobotics/rsl_rl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/leggedrobotics/rsl_rl
- Repository: https://github.com/leggedrobotics/rsl_rl

## Overview

##### Main Architectural Components
- rsl_rl Package: This is the heart of the library. Break it down into its submodules:
- Algorithms: Contains the implementation of RL algorithms (e.g., PPO, distillation).
- Env: Manages environment-related functionalities and wrappers (e.g., vectorized environments).
- Modules: Implements core building blocks like different variants of actor-critic networks, normalizers, and student-teacher components.
- Networks: Houses network components and memory modules used for high-level network operations.
- Runners: Contains the training loop logic; this is where the interaction between the environment, algorithm, and network happens.
- Storage: Provides mechanisms to store rollout experiences, essential for on-policy training.
- Utils: Includes helper functions, logging utilities for Neptune and Weights & Biases, and other general-purpose tools.

- Config and Setup:
- The “config” folder (with dummy_config.yaml) defines example configuration settings.
- setup.py, pyproject.toml, and various configuration and license files indicate deployment flexibility and external dependencies.

- Code Quality & Contribution Tools:
- Files such as .flake8, .pre-commit-config.yaml, and gitrelated files are setup for ensuring code quality but are not the primary runtime logic. These can be noted as auxiliary support for maintainability.

- External Integrations:
- Logging frameworks (as mentioned in the README).
- Integration with external simulation environments (e.g., Isaac Lab and Legged Gym).

##### Relationships and Data Flow
- Data Flow/Interaction:
- The Runner component orchestrates the training cycle by interacting with the environment (Env) to collect experiences.
- Collected experiences are stored in the Storage subsystem, which the algorithm components then use for policy updates.
- The Algorithm layer utilizes various modules (Actor-Critic, RND, Student-Teacher) as well as network components for forward passes and backpropagation.
- The Utils module supports auxiliary functions like normalization, logging (via integrated logging frameworks), and metric computation.

- Control Flow:
- Configurations are read from the config folder and are injected at various points (e.g., algorithm parameters, environment settings).
- The training loop (Runner) continuously fetches environment data and passes it through the appropriate algorithm and network components.
- Logging and visualization channels are updated throughout the training process, enabling real-time monitoring.

##### Architectural Patterns and Technologies
- Modular Design: The clear separation into algorithms, runners, storage, and utilities indicates a modular approach; each module has a single responsibility.
- Pipeline Structure: The interaction between environment simulation, data storage, algorithmic decision-making, and logging forms a sequential pipeline typical of RL systems.
- GPU-centric Operations: Although not explicitly detailed in the file structure, the README emphasizes full GPU utilization, which might indicate extensive use of libraries such as PyTorch (common in RL, even if not directly referenced by name).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top Layer: Config & External Integrations
    subgraph "Config & External Integrations"
        config["Config & Setup"]:::config
        tb["TensorBoard"]:::external
        wandb_ext["Weights&Biases"]:::external
        neptune_ext["Neptune"]:::external
        sim["Isaac Lab/Legged Gym"]:::external
    end

    %% Core Layer: rsl_rl Package
    rsl_rl_pkg["rsl_rl Package"]:::core

    %% Submodules within rsl_rl Package
    subgraph "Algorithms"
        algo_ppo["Algorithm: PPO"]:::algorithm
        algo_dist["Algorithm: Distillation"]:::algorithm
    end

    subgraph "Env"
        env_vec["Env: Vec Env"]:::env
    end

    subgraph "Modules"
        ac["Module: Actor-Critic"]:::module
        ac_rec["Module: Actor-Critic Recurrent"]:::module
        normalizer["Module: Normalizer"]:::module
        rnd["Module: RND"]:::module
        st["Module: Student-Teacher"]:::module
        st_rec["Module: Student-Teacher Recurrent"]:::module
    end

    subgraph "Networks"
        mem["Networks: Memory"]:::network
    end

    subgraph "Runners"
        runner["Runner: On-Policy Runner"]:::runner
    end

    subgraph "Storage"
        storage["Storage: Rollout Storage"]:::storage
    end

    subgraph "Utils"
        utils_neptune["Utils: Neptune"]:::utils
        utils_wandb["Utils: W&B"]:::utils
        utils_misc["Utils: Misc"]:::utils
    end

    %% Connections: Config to Core Package
    config -->|"configures"| rsl_rl_pkg

    %% Core Package Initialization to Submodules
    rsl_rl_pkg -->|"init"| algo_ppo
    rsl_rl_pkg -->|"init"| env_vec
    rsl_rl_pkg -->|"init"| ac
    rsl_rl_pkg -->|"init"| mem
    rsl_rl_pkg -->|"init"| runner
    rsl_rl_pkg -->|"init"| storage
    rsl_rl_pkg -->|"init"| utils_neptune

    %% RL Workflow: Training Loop and Data Flow
    runner -->|"collects_experience"| env_vec
    runner -->|"stores_data"| storage
    storage -->|"feeds_data"| algo_ppo
    runner -->|"policy_update"| algo_dist

    %% Algorithm operations
    algo_ppo -->|"forward_pass"| mem
    algo_ppo -->|"uses"| ac
    algo_ppo -->|"exploration"| rnd
    algo_dist -->|"teacher_policy"| st
    algo_dist -->|"teacher_policy"| st_rec

    %% Logging from Runner
    runner -->|"logs"| utils_misc
    utils_misc -->|"logs_to"| tb
    utils_misc -->|"logs_to"| wandb_ext
    utils_misc -->|"logs_to"| neptune_ext

    %% External Simulation Integration
    env_vec -->|"integrates"| sim

    %% Class Definitions for Colors and Styles
    classDef config fill:#ffcc00,stroke:#333,stroke-width:2px;
    classDef external fill:#99ccff,stroke:#333,stroke-width:2px;
    classDef core fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef algorithm fill:#ffdddd,stroke:#d33,stroke-width:2px;
    classDef env fill:#ddffdd,stroke:#3d3,stroke-width:2px;
    classDef module fill:#ffffcc,stroke:#cc0,stroke-width:2px;
    classDef network fill:#ffccff,stroke:#933,stroke-width:2px;
    classDef runner fill:#d9e1ff,stroke:#5d7,stroke-width:2px;
    classDef storage fill:#ffe6cc,stroke:#a66,stroke-width:2px;
    classDef utils fill:#e6e6fa,stroke:#666,stroke-width:2px;

    %% Click Events for Component Mapping
    click rsl_rl_pkg "https://github.com/leggedrobotics/rsl_rl/tree/main/rsl_rl/"
    click algo_ppo "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/algorithms/ppo.py"
    click algo_dist "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/algorithms/distillation.py"
    click env_vec "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/env/vec_env.py"
    click ac "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/modules/actor_critic.py"
    click ac_rec "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/modules/actor_critic_recurrent.py"
    click normalizer "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/modules/normalizer.py"
    click rnd "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/modules/rnd.py"
    click st "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/modules/student_teacher.py"
    click st_rec "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/modules/student_teacher_recurrent.py"
    click mem "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/networks/memory.py"
    click runner "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/runners/on_policy_runner.py"
    click storage "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/storage/rollout_storage.py"
    click utils_neptune "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/utils/neptune_utils.py"
    click utils_wandb "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/utils/wandb_utils.py"
    click utils_misc "https://github.com/leggedrobotics/rsl_rl/blob/main/rsl_rl/utils/utils.py"
    click config "https://github.com/leggedrobotics/rsl_rl/blob/main/config/dummy_config.yaml"
```
