How leggedrobotics/rsl_rl works
Main Architectural Components
• rsl_rl Package: This is the heart of the library. Break it down into its submodules:
• Algorithms: Contains the implementation of RL algorithms (e.g., PPO, distillation).
• Env: Manages environment-related functionalities and wrappers (e.g., vectorized environments).
• Modules: Implements core building blocks like different variants of actor-critic networks, normalizers, and student-teacher components.
• Networks: Houses network components and memory modules used for high-level network operations.
• Runners: Contains the training loop logic; this is where the interaction between the environment, algorithm, and network happens.
• Storage: Provides mechanisms to store rollout experiences, essential for on-policy training.
• Utils: Includes helper functions, logging utilities for Neptune and Weights & Biases, and other general-purpose tools.
• Config and Setup:
• The “config” folder (with dummy_config.yaml) defines example configuration settings.
• setup.py, pyproject.toml, and various configuration and license files indicate deployment flexibility and external dependencies.
• Code Quality & Contribution Tools:
• Files such as .flake8, .pre-commit-config.yaml, and gitrelated files are setup for ensuring code quality but are not the primary runtime logic. These can be noted as auxiliary support for maintainability.
• External Integrations:
• Logging frameworks (as mentioned in the README).
• Integration with external simulation environments (e.g., Isaac Lab and Legged Gym).
Relationships and Data Flow
• Data Flow/Interaction:
• The Runner component orchestrates the training cycle by interacting with the environment (Env) to collect experiences.
• Collected experiences are stored in the Storage subsystem, which the algorithm components then use for policy updates.
• The Algorithm layer utilizes various modules (Actor-Critic, RND, Student-Teacher) as well as network components for forward passes and backpropagation.
• The Utils module supports auxiliary functions like normalization, logging (via integrated logging frameworks), and metric computation.
• Control Flow:
• Configurations are read from the config folder and are injected at various points (e.g., algorithm parameters, environment settings).
• The training loop (Runner) continuously fetches environment data and passes it through the appropriate algorithm and network components.
• Logging and visualization channels are updated throughout the training process, enabling real-time monitoring.
Architectural Patterns and Technologies
• Modular Design: The clear separation into algorithms, runners, storage, and utilities indicates a modular approach; each module has a single responsibility.
• Pipeline Structure: The interaction between environment simulation, data storage, algorithmic decision-making, and logging forms a sequential pipeline typical of RL systems.
• GPU-centric Operations: Although not explicitly detailed in the file structure, the README emphasizes full GPU utilization, which might indicate extensive use of libraries such as PyTorch (common in RL, even if not directly referenced by name).