How katerakelly/oyster works
Step 1: Identify the Main Components
• Experiment Launcher & Config Manager: The scripts “launch.py” and “launch_experiment.py” use configuration files in the “configs” directory. These represent the entry point into the system, triggering training runs and experiment setups.
• Environment Definitions: Inside the “rlkit/envs” folder, you see environment scripts (e.g., ant.py, half_cheetah.py, walker wrappers) and related XML assets for MuJoCo simulation. These form the simulated worlds where the RL agent learns.
• RL Algorithm & Policy Modules: The heart of the implementation is in “rlkit/core”, “rlkit/policies”, “rlkit/samplers” and “rlkit/torch/sac”, where the RL algorithms (including SAC for continuous control) and associated policies and sampling methods are implemented.
• Data Management & Logging: “rlkit/data_management” handles replay buffers, normalization, and path building. The “rlkit/core/logger.py” is used for logging both progress and experimental statistics.
• Deep Learning Infrastructure: The “rlkit/torch” directory includes modules that define networks, helper functions for pytorch, and support for GPU acceleration.
• Additional Modules:
• “rand_param_envs” module: Provides environments with randomized parameters for tasks like Walker and Hopper.
• Docker Configuration: The “docker” folder’s Dockerfile and supportive scripts ensure that experiments can run in a containerized, GPU-enabled environment.
• Visualization & Evaluation: Extra scripts like “sim_policy.py” for evaluating and visualizing learned policies and a Jupyter notebook (“sparse-point-robot-vis.ipynb”) for interactive exploration complete the picture.
Step 2: Capture Relationships and Data Flow
• The launch scripts load experiment configurations (from “configs”) that specify which simulation environment (MuJoCo-based environments in “rlkit/envs” and “rand_param_envs”) and RL algorithm (from modules in “rlkit/torch/sac” and “rlkit/core/rl_algorithm.py”) to use.
• The Agent interacts with the Environment via samplers (in “rlkit/samplers”) that collect experience.
• Experience is stored and processed using data management components (replay buffers and normalizers).
• The Policy (defined in “rlkit/policies”) is updated by the RL algorithm’s learning procedure.
• During training, logging (via “rlkit/core/logger.py”) continuously outputs metrics that can later be visualized using external tools like “viskit”.
• Docker encapsulates the entire system for deployment on GPU-supported machines, while ensuring that external dependencies (e.g., MuJoCo) are correctly handled.
Step 3: Architectural Patterns & Design Principles
• Modularity: The project uses a layered architecture. Components are neatly decomposed into configuration, environment, data management, RL algorithms, and deep learning support.
• Separation of Concerns: There is a clear boundary between experiment setup (launchers, configs), simulation (envs, rand_param_envs), learning (core, torch, policies, samplers), and evaluation (simulation/visualization scripts).
• Extensibility: The per-environment setup and modular RL algorithm code allow for plugging in alternative policies, replay buffers, or even an encoder (as noted in the TODO for an RNN encoder and optional convolutional encoder).
For this meta-RL project:
a. Create nodes/blocks that represent:
• Experiment Launcher (launch.py & launch_experiment.py)
• Configuration Manager (configs/ JSON and Python files)
• Environment Simulator (rlkit/envs, rand_param_envs, and related XML assets)
• Learning Core (rlkit/core and rlkit/torch modules, including SAC agent)
• Data Management (replay buffers, logging, and normalization)
• Policy & Sampler (rlkit/policies, rlkit/samplers)
• Visualization/Evaluation (sim_policy.py, notebook)
• Docker Container (docker folder with Dockerfile and environment.yml)
• The flow from configurations to the Experiment Launcher.
• The interaction between the RL agent (learning core, policies, and samplers) and the Environment.
• The data flow from experience collection through replay buffers to RL updates.
• Feedback loops (e.g., logging and network snapshots during training).
• A third for the learning algorithms (core, torch, policies).
d. Text labels for each block:
• Not just generic names but indicate key roles (e.g., “MuJoCo Environment Wrapper”, “SAC Agent & RL Update Logic”, “Replay Buffer – Data Management”).
e. External Dependencies: