# google-deepmind/disco_rl architecture

> Architecture diagram of the GitHub repository google-deepmind/disco_rl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google-deepmind/disco_rl
- Repository: https://github.com/google-deepmind/disco_rl

## Overview

##### Project Type & Purpose
- Type: Open-source RL library/harness with meta‐training and meta‐evaluation workflows.
- Purpose: Provide a flexible API for (a) evaluating the “Disco103” discovered RL update rule and (b) meta‐learning new RL update rules from scratch or checkpoints.

##### File Structure
- Top-level:
– colabs/ : Jupyter notebooks for evaluation & meta-training
– disco_rl/: Python package
– pyproject.toml, README, LICENSE, etc.
- Under disco_rl/:
– environments/: interface + Catch implementations + wrappers (batch/jittable)
– networks/: Haiku MLP/LSTM modules
– update_rules/: plug-in architecture for Disco, actor-critic, policy-gradient, transforms, and pre-trained weights
– value_fns/: value‐function utilities
– agent.py: generic RL agent using the update_rule API
– optimizers.py, types.py, utils.py: core harness utilities

##### README Insights
- Confirms split into meta‐evaluation (eval.ipynb) vs. meta‐training (meta_train.ipynb).
- Notes dependencies: JAX, Haiku, NumPy.
- Update_rules defines a uniform interface that agent.py consumes.

##### Main Components & Interactions
a. Components
- Client Layer: colabs notebooks (eval, meta_train)
- API Layer: Agent (agent.py)
- Core Modules:
– Environments (step, reset, wrappers for batching/JIT)
– Networks (Haiku models: nets.py, action_models.py, meta_nets.py)
– UpdateRules (disco.py, actor_critic.py, policy_gradient.py + base definitions + input transforms + pre-learned weights)
– ValueFunctions (value_fn interface + utilities)
– Optimizers (optimizers.py)
- Utilities: types.py, utils.py
- External Dependencies: JAX, Haiku, NumPy
- Data Artifact: disco_103.npz (pre-trained weights)

b. Relationships & Data Flow
1. Notebook instantiates Agent with a chosen UpdateRule and Network architecture.
2. Agent initializes parameters (possibly from disco_103.npz).
3. In each training step:
– Agent queries Environment(s) via wrappers → receives observations, rewards.
– Agent forwards observations through Networks to get actions/values.
– UpdateRule computes parameter updates using optimizer, networks, and value_fns.
– Optimizer applies gradients to Networks.
– Loop continues.
4. In meta-training, the outer loop updates the UpdateRule parameters via meta-gradients.

c. Architectural Patterns & Principles
- Plug-in Architecture: update_rules all adhere to a common interface.
- Separation of Concerns: environments, networks, update logic, and optimization are modular.
- Functional JAX Style: pure functions, explicit parameter passing.
- Batch & JIT wrappers for performance scalability.

d. Key Technologies
- JAX for automatic differentiation and JIT compilation.
- Haiku for neural‐network definitions.
- NumPy for data handling.

##### Final Advice
- Data flows at both the RL‐step level (inner loop) and meta‐loop (outer loop).
- Both CPU‐based and jittable environments under the “Environment” component.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        CL1["eval.ipynb"]:::client
        CL2["meta_train.ipynb"]:::client
    end

    %% API Layer
    subgraph "API Layer"
        Agent["Agent (agent.py)"]:::core
    end

    %% Core Modules
    subgraph "Core Modules"
        subgraph "Environments"
            EnvBase["base.py"]:::core
            EnvCatch["catch.py"]:::core
            EnvJit["jittable_envs.py"]:::core
            subgraph "Wrappers"
                WrapBatch["batched_env.py"]:::core
                WrapBatchJit["batched_jittable_env.py"]:::core
                WrapSingle["single_stream_env.py"]:::core
            end
        end
        subgraph "Networks"
            NetCore["nets.py"]:::core
            NetAction["action_models.py"]:::core
            NetMeta["meta_nets.py"]:::core
        end
        subgraph "UpdateRules"
            URBase["base.py"]:::core
            URDisco["disco.py"]:::core
            URAC["actor_critic.py"]:::core
            URPG["policy_gradient.py"]:::core
            URTrans["input_transforms.py"]:::core
            URWeights["disco_103.npz"]:::data
        end
        subgraph "ValueFunctions"
            VFInt["value_fn.py"]:::core
            VFUtil["value_utils.py"]:::core
        end
        Optimizer["optimizers.py"]:::core
        subgraph "Utilities"
            Types["types.py"]:::core
            Utils["utils.py"]:::core
        end
    end

    %% External Dependencies
    subgraph "External Dependencies"
        JAX["JAX"]:::external
        Haiku["Haiku"]:::external
        NumPy["NumPy"]:::external
    end

    %% Data Flows
    CL1 -->|uses| Agent
    CL2 -->|uses| Agent

    Agent -->|queries| EnvBase
    Agent --> EnvCatch
    Agent --> EnvJit
    EnvBase -->|wraps| WrapBatch
    EnvBase --> WrapBatchJit
    EnvBase --> WrapSingle
    EnvBase -->|observations| Agent

    Agent -->|forward| NetCore
    Agent --> NetAction
    Agent --> NetMeta

    Agent -->|invokes| URBase
    URBase --> URDisco
    URBase --> URAC
    URBase --> URPG
    URBase --> URTrans
    URDisco --> URWeights
    URBase -->|computes| Optimizer
    Optimizer -->|updates| NetCore
    Optimizer --> NetAction
    Optimizer --> NetMeta

    URBase --> VFInt
    VFInt --> VFUtil

    Agent --> Types
    Agent --> Utils
    URBase --> Types
    URBase --> Utils

    %% External dependencies
    Agent --> JAX
    NetCore --> Haiku
    NetAction --> Haiku
    NetMeta --> Haiku
    CL1 --> NumPy
    CL2 --> NumPy
    URWeights --> NumPy

    %% Click Events
    click CL1 "https://github.com/google-deepmind/disco_rl/blob/main/colabs/eval.ipynb"
    click CL2 "https://github.com/google-deepmind/disco_rl/blob/main/colabs/meta_train.ipynb"
    click Agent "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/agent.py"
    click EnvBase "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/environments/base.py"
    click EnvCatch "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/environments/catch.py"
    click EnvJit "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/environments/jittable_envs.py"
    click WrapBatch "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/environments/wrappers/batched_env.py"
    click WrapBatchJit "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/environments/wrappers/batched_jittable_env.py"
    click WrapSingle "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/environments/wrappers/single_stream_env.py"
    click NetCore "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/networks/nets.py"
    click NetAction "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/networks/action_models.py"
    click NetMeta "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/networks/meta_nets.py"
    click URBase "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/update_rules/base.py"
    click URDisco "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/update_rules/disco.py"
    click URAC "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/update_rules/actor_critic.py"
    click URPG "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/update_rules/policy_gradient.py"
    click URTrans "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/update_rules/input_transforms.py"
    click URWeights "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/update_rules/weights/disco_103.npz"
    click VFInt "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/value_fns/value_fn.py"
    click VFUtil "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/value_fns/value_utils.py"
    click Optimizer "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/optimizers.py"
    click Types "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/types.py"
    click Utils "https://github.com/google-deepmind/disco_rl/blob/main/disco_rl/utils.py"

    %% Styles
    classDef client fill:#a2f,stroke:#333,color:#000
    classDef core fill:#2af,stroke:#333,color:#000
    classDef external fill:#ccc,stroke:#333,color:#000
    classDef data fill:#fea,stroke:#333,color:#000
```
