# google-deepmind/acme architecture

> Architecture diagram of the GitHub repository google-deepmind/acme, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google-deepmind/acme
- Repository: https://github.com/google-deepmind/acme

## Overview

##### Project type and purpose
- Acme is an open-source Python framework for reinforcement learning (RL), providing modular “building blocks” (agents, adders, loops, datasets) with both JAX and TensorFlow backends.
- Its goal is to offer reference implementations and strong baselines that scale from single-process experiments to distributed training.

##### File structure and core modules
- Top-level packages:
– acme/adders: components that insert experience into replay (Reverb)
– acme/agents: core Agent interface and two subtrees (jax, tf) containing per-algorithm builders, learners, networks, etc.
– acme/datasets: wrappers and iterators to read from replay buffers
– acme/environment_loop & acme/environment_loops: the driving loop that steps the env, applies adders, and calls the agent
– acme/jax & acme/tf: backend-specific utilities, experiment runners, saving, snapshotting, inference server (JAX)
– acme/wrappers: Gym/DM-Control wrappers to canonicalize observations and actions
– acme/utils: logging, metrics, observers, paths, asynchronous helpers
– acme/testing: fakes and test utilities
- Support directories:
– examples/: runnable scripts and notebooks demonstrating usage
– .github/, CI workflows, linting, packaging files

##### README insights
- Acme strives for simple, readable, composable Agents that work single-stream or distributed.
- It uses external services (Reverb) and supports both JAX and TensorFlow.
- Examples and tutorials show how to wire components together.

##### Main components
a. User/Experiment Layer
– Example scripts / notebooks / CLI (examples/, acme/jax/experiments)
b. Environment Layer
– External environments (Gym, dm_control, bsuite)
– acme/wrappers to adapt env API
c. Environment Loop
– acme/environment_loop stepping: calls actor to get action, steps env, sends transition to adder
d. Adder & Replay Buffer
– acme/adders packages writing to a Reverb server (external process)
e. Dataset Pipeline
– acme/datasets reading from Reverb, creating Python or TF/JAX iterators
f. Agent
– Builder (configures networks, learner, actor)
– Learner (consumes dataset, computes gradients via JAX or TF)
– Actor (holds up-to-date policy, performs inference)
g. Backend Abstractions
– acme/jax and acme/tf modules: networks, losses, savers, snapshotter, variable_utils
– Inference server (JAX) to serve actor policy to remote evaluators
h. Utilities & Logging
– acme/utils/loggers, observers, snapshotter: record metrics, save checkpoints
i. Multiagent Support (optional)
– acme/multiagent utilities and types

##### Relationships and data flow
– User/Experiment triggers AgentBuilder → spawns Actor & Learner
– Actor ↔ Learner share parameters through snapshotter or direct in-memory variables
– Actor drives EnvironmentLoop:
EnvironmentLoop → Wrapper → Env.step() → Observation, Reward → Adder
– Adder pushes to Reverb server
– Learner pulls batches from Reverb via Dataset → computes updates → updates parameters
– Actor polls for new parameters (via snapshotter or InferenceServer)
– Logging hooks record metrics at key points (env rewards, learner losses)

##### Architectural patterns and design principles
- Modularity: clear separation of concerns (Env, Loop, Adder, Replay, Learner, Actor)
- Actor-Learner separation: decouples data collection from policy learning
- Backend abstraction: common interfaces in acme/agents with separate JAX and TF implementations
- Builder pattern: algorithm-specific builders configure and assemble components
- Service decoupling: Reverb as an external replay service enables distributed setups

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User/Experiment Layer
    subgraph "User / Experiment Layer"
        direction TB
        Examples["Examples / Scripts / Notebooks"]:::user
    end

    %% Environment Layer
    subgraph "Environment Layer"
        direction TB
        EnvExt["External Environments (Gym, dm_control, bsuite)"]:::external
        Wrapper["Wrappers (acme/wrappers)"]:::core
    end

    %% Core RL Loop
    subgraph "Core RL Loop" 
        direction TB
        EnvLoop["EnvironmentLoop"]:::core
        Adder["Adder (acme/adders)"]:::core
        ReverbServer["Reverb Server"]:::external
        Dataset["Dataset Pipeline"]:::core
        Learner["Learner"]:::core
        Actor["Actor"]:::core
        Builder["Agent Builder"]:::core
    end

    %% Backend Abstractions
    subgraph "Backend Abstractions" 
        direction TB
        JAX["JAX Backend (acme/jax)"]:::backend
        TF["TensorFlow Backend (acme/tf)"]:::backend
        InferenceServer["Inference Server"]:::backend
    end

    %% Utilities & Logging
    subgraph "Utilities & Logging"
        direction TB
        Utils["Utilities & Logging (acme/utils)"]:::utils
        Snapshotter["Snapshotter"]:::utils
    end

    %% Connections
    Examples -->|"launches"| Builder
    Builder -->|"configures"| Actor
    Builder -->|"configures"| Learner
    Actor -->|"drives"| EnvLoop
    EnvLoop -->|"- actions ->"| EnvExt
    EnvExt -->|"observations, rewards"| Wrapper
    Wrapper -->|"adapted transitions"| Adder
    Adder -->|"push transitions ->"| ReverbServer
    Learner -->|"pull batches <-"| Dataset
    Dataset -->|"batches ->"| Learner
    Learner -->|"updates params ->"| Snapshotter
    Snapshotter -->|"sync params ->"| Actor
    InferenceServer -->|"serves policy ->"| Actor
    JAX -->|"provides nets, losses"| Learner
    TF -->|"provides nets, losses"| Learner
    Utils -->|"logging & metrics"| EnvLoop
    Utils -->|"logging & metrics"| Learner

    %% Click Events
    click Examples "https://github.com/google-deepmind/acme/tree/master/examples/"
    click Wrapper "https://github.com/google-deepmind/acme/tree/master/acme/wrappers/"
    click EnvLoop "https://github.com/google-deepmind/acme/blob/master/acme/environment_loop.py"
    click Adder "https://github.com/google-deepmind/acme/blob/master/acme/adders/base.py"
    click ReverbServer "https://github.com/google-deepmind/acme/tree/master/acme/adders/reverb/"
    click Dataset "https://github.com/google-deepmind/acme/tree/master/acme/datasets/"
    click Builder "https://github.com/google-deepmind/acme/blob/master/acme/agents/agent.py"
    click JAX "https://github.com/google-deepmind/acme/tree/master/acme/jax/"
    click TF "https://github.com/google-deepmind/acme/tree/master/acme/tf/"
    click InferenceServer "https://github.com/google-deepmind/acme/blob/master/acme/jax/inference_server.py"
    click Utils "https://github.com/google-deepmind/acme/tree/master/acme/utils/"
    click Snapshotter "https://github.com/google-deepmind/acme/blob/master/acme/jax/snapshotter.py"

    %% Styles
    classDef user fill:#e3f2fd,stroke:#90caf9,color:#0d47a1
    classDef external fill:#fff3e0,stroke:#ffb74d,color:#e65100
    classDef core fill:#e8f5e9,stroke:#66bb6a,color:#1b5e20
    classDef backend fill:#e1f5fe,stroke:#29b6f6,color:#01579b
    classDef utils fill:#eeeeee,stroke:#bdbdbd,color:#424242
```
