# tensorflow/agents architecture

> Architecture diagram of the GitHub repository tensorflow/agents, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tensorflow/agents
- Repository: https://github.com/tensorflow/agents

## Overview

##### Project Type and Purpose
- TF-Agents is not a standalone application but a modular Python library/framework for building, training, and evaluating Reinforcement Learning (RL) and Contextual Bandits algorithms on top of TensorFlow.
- It provides abstractions for Environments, Agents (Policies + Learning), Replay Buffers, Drivers, Networks, Metrics, and Training loops.

##### High-Level Module Structure
Top-level directories under `tf_agents/`:
- environments – wraps and batches Gym/DM Control/Atari/etc.
- specs – describes observations/actions via ArraySpec/TensorSpec.
- trajectories – holds time-step and transition data structures.
- policies – implements action selection logic.
- networks – neural network building blocks.
- agents – core RL & Bandit algorithms (DQN, PPO, SAC, CQL, Bandits, etc.).
- replay_buffers – experience storage (in-memory, Reverb, RLDS).
- drivers – rollouts: stepping environment(s) to collect data.
- train – orchestration of actor/learner loops, triggers, metrics.
- metrics/eval – logging, performance measurement utilities.
- distributions – probability distributions for policies.
- utils – shared helper functions (nest utils, session utils, etc.).
- keras_layers – custom Keras layers used by networks.
- examples & experimental – end-to-end scripts and distributed variants.

##### Core Architectural Patterns
- Layered, plugin-style design: each module defines clear interfaces (e.g., `PyEnvironment`, `TFPolicy`, `Network`, `Agent`).
- Separation of concerns: Environments generate observations; Drivers + Policies produce actions; Replay Buffers store transitions; Agents consume batches to update Networks; Train/Actor/Learner glue everything together.
- Optional external service: Reverb (for scalable replay), configured via `tf_agents/replay_buffers/reverb_replay_buffer.py`.

##### Relationships and Data Flow
- Environment → Driver: the Driver steps the Environment, producing Trajectories.
- Driver → Replay Buffer: collected Trajectories are pushed into the Buffer.
- Agent (via Learner) ↔ Replay Buffer: the Learner samples Batches to update the Agent’s Network/Policy.
- Agent → Policy: after update, the Agent exposes a Policy for action selection.
- Trainer/Actor → Policy + Driver: in distributed setups, Actors pull latest Policy, drive Environments, feed data back.
- Metrics/Eval subscribe to Trajectories and Agent updates to compute performance.

- Core Modules (blue): Environments, Trajectories, Specs
- Algorithm Modules (green): Agents, Policies, Networks
- Data Infrastructure (orange): Replay Buffers, Drivers
- Orchestration (purple): train.Actor, train.Learner, Triggers
- Utilities (gray): metrics, eval, utils, distributions
- External Services (red): TensorFlow runtime, Reverb server (optional)

- Env → Driver → Buffer
- Buffer → Learner
- Learner → Network / Policy
- Policy → Driver (closed loop)
- Actor ↔ Learner in distributed mode

c. Layering (vertical stack):
Top: User scripts / Examples
Next: train / experimental orchestrators
Next: Agents + Policies + Networks
Next: Drivers + Replay Buffers
Next: Environments + Specs + Trajectories
Bottom: TensorFlow + Reverb

- Agents depend on networks, policies, distributions, utils.
- Drivers depend on environments and policies.
- Replay buffers optionally depend on Reverb.
- Metrics/utilities cross-cut all layers.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Scripts
    subgraph "User Scripts / Examples"
        EX["Examples"]:::external
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        A1["train: Actor & Learner"]:::orchestration
    end

    %% Algorithm Modules
    subgraph "Algorithm Modules"
        Agents["Agents\n(DQN, PPO, SAC, CQL, Bandits)"]:::algorithm
        Policies["Policies\n(TFPolicy)"]:::algorithm
        Networks["Networks"]:::algorithm
    end

    %% Data Infrastructure
    subgraph "Data Infrastructure"
        Drivers["Drivers"]:::data
        Replay["Replay Buffers\n(TFReplayBuffer)"]:::data
    end

    %% Core Modules
    subgraph "Core Modules"
        Env["Environments\n(PyEnvironment)"]:::core
        Specs["Specs\n(ArraySpec/TensorSpec)"]:::core
        Traj["Trajectories\n(TimeStep)"]:::core
    end

    %% Utilities
    subgraph "Utilities"
        Metrics["Metrics"]:::utils
        Eval["Evaluation"]:::utils
        Utils["Utils"]:::utils
        Distributions["Distributions"]:::utils
        Keras["Keras Layers"]:::utils
    end

    %% External Services
    subgraph "External Services"
        TF["TensorFlow Runtime"]:::external
        Reverb["Reverb Server"]:::external
    end

    %% Primary Data Flow
    EX -->|uses| A1
    A1 -->|orchestrates| Agents
    A1 -->|triggers| Drivers
    Agents -->|uses| Policies
    Agents -->|uses| Networks
    Drivers -->|collects| Traj
    Traj -->|pushes to| Replay
    Replay -->|samples for| Agents
    Policies -->|selects actions for| Drivers
    Env -->|stepped by| Drivers

    %% Closed Loop
    Policies -->|action loop| Drivers

    %% Cross-Cutting Metrics
    Traj -->|feeds| Metrics
    Agents -->|updates| Metrics
    Agents -->|evaluated by| Eval

    %% Dashed Dependencies
    Agents -.->|depends on| Utils
    Agents -.->|depends on| Distributions
    Drivers -.->|depends on| Env
    Replay -.->|optional Reverb| Reverb
    Env -.->|runs on| TF
    Policies -.->|runs on| TF
    Networks -.->|runs on| TF
    Agents -.->|runs on| TF
    Replay -.->|runs on| TF

    %% Click Events
    click EX "https://github.com/tensorflow/agents/tree/master/tf_agents/examples/"
    click A1 "https://github.com/tensorflow/agents/tree/master/tf_agents/train/"
    click Env "https://github.com/tensorflow/agents/tree/master/tf_agents/environments/"
    click Specs "https://github.com/tensorflow/agents/tree/master/tf_agents/specs/"
    click Traj "https://github.com/tensorflow/agents/tree/master/tf_agents/trajectories/"
    click Policies "https://github.com/tensorflow/agents/tree/master/tf_agents/policies/"
    click Networks "https://github.com/tensorflow/agents/tree/master/tf_agents/networks/"
    click Agents "https://github.com/tensorflow/agents/tree/master/tf_agents/agents/"
    click Replay "https://github.com/tensorflow/agents/tree/master/tf_agents/replay_buffers/"
    click Drivers "https://github.com/tensorflow/agents/tree/master/tf_agents/drivers/"
    click Metrics "https://github.com/tensorflow/agents/tree/master/tf_agents/metrics/"
    click Eval "https://github.com/tensorflow/agents/tree/master/tf_agents/eval/"
    click Distributions "https://github.com/tensorflow/agents/tree/master/tf_agents/distributions/"
    click Utils "https://github.com/tensorflow/agents/tree/master/tf_agents/utils/"
    click Keras "https://github.com/tensorflow/agents/tree/master/tf_agents/keras_layers/"
    click Reverb "https://github.com/tensorflow/agents/blob/master/tf_agents/replay_buffers/reverb_replay_buffer.py"

    %% Styles
    classDef core       fill:#cce5ff,stroke:#333,stroke-width:1px;
    classDef algorithm  fill:#d4edda,stroke:#333,stroke-width:1px;
    classDef data       fill:#ffe5b4,stroke:#333,stroke-width:1px;
    classDef orchestration fill:#e2ccff,stroke:#333,stroke-width:1px;
    classDef utils      fill:#e0e0e0,stroke:#333,stroke-width:1px;
    classDef external   fill:#f8d7da,stroke:#333,stroke-width:1px;
```
