How graliuce/sgcrl works
This project is a reinforcement learning (RL) application designed to train agents using a contrastive learning setup – specifically, it implements the Single-goal Contrastive RL (SGCRL) method. The project is tailored toward robotics or simulated robotics tasks (e.g., sawyer_bin, sawyer_box, sawyer_peg environments) and leverages contrastive methods to guide the training process.
Main Components of the System
• Entry Point & Execution Flow:
• The main entry script is “lp_contrastive.py” which serves as the launcher to initiate training.
• Other executable scripts like “default.py”, “distributional.py”, “env_utils.py”, and “point_env.py” indicate various runtime utilities and environment setups.
• Core RL Modules (inside the “contrastive” folder):
• agents.py: Defines the RL agents and their behavior.
• learning.py: Contains the learning loop and core RL algorithm logic (contrastive reinforcement learning).
• networks.py: Houses the definitions of neural network architectures employed by the agents.
• config.py: Stores configuration details that tailor the agent behavior and experimental setups.
• builder.py: Likely responsible for constructing the components (agents, networks, etc.) based on the configuration.
• distributed_layout.py: Implements support for distributed and possibly parallel execution across GPUs or CPUs.
• utils.py: Provides helper functions or common utilities used throughout the project.
• Additional Components:
• Training video resources: Videos (e.g., sawyer_bin.mp4, sawyer_box.mp4, etc.) and zipped training videos in the “training_videos” folder. These resources help demonstrate training results or provide visual feedback of the experiments.
• Environment wrappers: Files like “env_utils.py” and “point_env.py” help integrate with simulation environments or task-specific settings.
Out Relationships and Interactions
• Control Flow:
• The entry point “lp_contrastive.py” initiates execution by parsing command-line flags (e.g., environment type, algorithm type, number of steps) as defined in the README.
• It loads configurations (from config.py) and uses the builder (builder.py) to instantiate core components such as agents and policy networks.
• Data Flow & Dependencies:
• The agent (implemented in agents.py and supported by learning.py) interacts with the environment (initialized via env_utils.py or point_env.py) to gather experiences.
• Neural network modules (networks.py) process state and action information, feeding into the learning loop where gradients and updates are computed.
• In distributed setups (handled by distributed_layout.py), multiple sub-components may communicate for parallel computations.
• External Dependencies & Integrations:
• External libraries (e.g., JAX, PyTorch, dm-haiku, optax) are integrated as outlined in the README. Their roles in acceleration (GPU support) and numerical computation must be clearly indicated.
• Mujoco is a key library for rendering and simulation physics, linked via environment setup instructions.
Architectural Patterns and Design Principles
• Modularity: The code is organized into clear modules (agent behavior, configuration, model building, and distributed learning), making separation of concerns a prime consideration.
• Configurability: The configuration file (config.py) and command-line flags ensure the design supports several environments and algorithms.
• Scalability: Support for distributed layouts suggests a design that scales computation across multiple cores or GPUs.
• Extensibility: With distinct components for agents, networks, and utilities, it’s easy to extend or swap out parts of the system.