How google-deepmind/acme works
Project type and purpose
• Acme is an open-source Python framework for reinforcement learning (RL), providing modular “building blocks” (agents, adders, loops, datasets) with both JAX and TensorFlow backends.
• Its goal is to offer reference implementations and strong baselines that scale from single-process experiments to distributed training.
File structure and core modules
• Top-level packages:
– acme/adders: components that insert experience into replay (Reverb)
– acme/agents: core Agent interface and two subtrees (jax, tf) containing per-algorithm builders, learners, networks, etc.
– acme/datasets: wrappers and iterators to read from replay buffers
– acme/environment_loop & acme/environment_loops: the driving loop that steps the env, applies adders, and calls the agent
– acme/jax & acme/tf: backend-specific utilities, experiment runners, saving, snapshotting, inference server (JAX)
– acme/wrappers: Gym/DM-Control wrappers to canonicalize observations and actions
– acme/utils: logging, metrics, observers, paths, asynchronous helpers
– acme/testing: fakes and test utilities
• Support directories:
– examples/: runnable scripts and notebooks demonstrating usage
– .github/, CI workflows, linting, packaging files
README insights
• Acme strives for simple, readable, composable Agents that work single-stream or distributed.
• It uses external services (Reverb) and supports both JAX and TensorFlow.
• Examples and tutorials show how to wire components together.
Main components
a. User/Experiment Layer
– Example scripts / notebooks / CLI (examples/, acme/jax/experiments)
b. Environment Layer
– External environments (Gym, dm_control, bsuite)
– acme/wrappers to adapt env API
c. Environment Loop
– acme/environment_loop stepping: calls actor to get action, steps env, sends transition to adder
d. Adder & Replay Buffer
– acme/adders packages writing to a Reverb server (external process)
e. Dataset Pipeline
– acme/datasets reading from Reverb, creating Python or TF/JAX iterators
f. Agent
– Builder (configures networks, learner, actor)
– Learner (consumes dataset, computes gradients via JAX or TF)
– Actor (holds up-to-date policy, performs inference)
g. Backend Abstractions
– acme/jax and acme/tf modules: networks, losses, savers, snapshotter, variable_utils
– Inference server (JAX) to serve actor policy to remote evaluators
h. Utilities & Logging
– acme/utils/loggers, observers, snapshotter: record metrics, save checkpoints
i. Multiagent Support (optional)
– acme/multiagent utilities and types
Relationships and data flow
– User/Experiment triggers AgentBuilder → spawns Actor & Learner
– Actor ↔ Learner share parameters through snapshotter or direct in-memory variables
– Actor drives EnvironmentLoop:
EnvironmentLoop → Wrapper → Env.step() → Observation, Reward → Adder
– Adder pushes to Reverb server
– Learner pulls batches from Reverb via Dataset → computes updates → updates parameters
– Actor polls for new parameters (via snapshotter or InferenceServer)
– Logging hooks record metrics at key points (env rewards, learner losses)
Architectural patterns and design principles
• Modularity: clear separation of concerns (Env, Loop, Adder, Replay, Learner, Actor)
• Actor-Learner separation: decouples data collection from policy learning
• Backend abstraction: common interfaces in acme/agents with separate JAX and TF implementations
• Builder pattern: algorithm-specific builders configure and assemble components
• Service decoupling: Reverb as an external replay service enables distributed setups