How google-deepmind/disco_rl works
Project Type & Purpose
• Type: Open-source RL library/harness with meta‐training and meta‐evaluation workflows.
• Purpose: Provide a flexible API for (a) evaluating the “Disco103” discovered RL update rule and (b) meta‐learning new RL update rules from scratch or checkpoints.
File Structure
• Top-level:
– colabs/ : Jupyter notebooks for evaluation & meta-training
– disco_rl/: Python package
– pyproject.toml, README, LICENSE, etc.
• Under disco_rl/:
– environments/: interface + Catch implementations + wrappers (batch/jittable)
– networks/: Haiku MLP/LSTM modules
– update_rules/: plug-in architecture for Disco, actor-critic, policy-gradient, transforms, and pre-trained weights
– value_fns/: value‐function utilities
– agent.py: generic RL agent using the update_rule API
– optimizers.py, types.py, utils.py: core harness utilities
README Insights
• Confirms split into meta‐evaluation (eval.ipynb) vs. meta‐training (meta_train.ipynb).
• Notes dependencies: JAX, Haiku, NumPy.
• Update_rules defines a uniform interface that agent.py consumes.
Main Components & Interactions
a. Components
• Client Layer: colabs notebooks (eval, meta_train)
• API Layer: Agent (agent.py)
• Core Modules:
– Environments (step, reset, wrappers for batching/JIT)
– Networks (Haiku models: nets.py, action_models.py, meta_nets.py)
– UpdateRules (disco.py, actor_critic.py, policy_gradient.py + base definitions + input transforms + pre-learned weights)
– ValueFunctions (value_fn interface + utilities)
– Optimizers (optimizers.py)
• Utilities: types.py, utils.py
• External Dependencies: JAX, Haiku, NumPy
• Data Artifact: disco_103.npz (pre-trained weights)
b. Relationships & Data Flow
1. Notebook instantiates Agent with a chosen UpdateRule and Network architecture.
2. Agent initializes parameters (possibly from disco_103.npz).
3. In each training step:
– Agent queries Environment(s) via wrappers → receives observations, rewards.
– Agent forwards observations through Networks to get actions/values.
– UpdateRule computes parameter updates using optimizer, networks, and value_fns.
– Optimizer applies gradients to Networks.
– Loop continues.
4. In meta-training, the outer loop updates the UpdateRule parameters via meta-gradients.
c. Architectural Patterns & Principles
• Plug-in Architecture: update_rules all adhere to a common interface.
• Separation of Concerns: environments, networks, update logic, and optimization are modular.
• Functional JAX Style: pure functions, explicit parameter passing.
• Batch & JIT wrappers for performance scalability.
d. Key Technologies
• JAX for automatic differentiation and JIT compilation.
• Haiku for neural‐network definitions.
• NumPy for data handling.
Final Advice
• Data flows at both the RL‐step level (inner loop) and meta‐loop (outer loop).
• Both CPU‐based and jittable environments under the “Environment” component.