# pytorch/rl architecture

> Architecture diagram of the GitHub repository pytorch/rl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pytorch/rl
- Repository: https://github.com/pytorch/rl

## Overview

TorchRL is a composable, PyTorch-native reinforcement-learning toolkit organized around TensorDict data. The principal workflow is policy–environment interaction, collection of trajectories, optional replay storage and sampling, then objective computation and optimization. The README also documents broad environment and transform support, distributed and recurrent workflows, model-based and multi-agent learning, and LLM components. Edges below follow the documented mental model; sampled source excerpts are incomplete, so relationships not established by the README or visible references are omitted.

Main entry points: [`torchrl/envs/common.py`](https://github.com/pytorch/rl/blob/main/torchrl/envs/common.py) (Environments), [`torchrl/trainers/trainers.py`](https://github.com/pytorch/rl/blob/main/torchrl/trainers/trainers.py) (Training orchestration), [`torchrl/collectors/collectors.py`](https://github.com/pytorch/rl/blob/main/torchrl/collectors/collectors.py) (Collectors), [`torchrl/modules/tensordict_module/actors.py`](https://github.com/pytorch/rl/blob/main/torchrl/modules/tensordict_module/actors.py) (Policy modules)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interaction["Interaction"]
  node_environments["Environments<br/>[common.py]"]
  node_transforms["Environment transforms<br/>[transforms.py]"]
  node_policies["Policy modules<br/>[actors.py]"]
  node_llm["LLM environments and policies<br/>[envs.py]"]
end

subgraph group_data["Data pipeline"]
  node_tensordict["TensorDict data"]
  node_specs["Tensor specs<br/>[tensor_specs.py]"]
  node_collectors["Collectors<br/>[collectors.py]"]
  node_replay[("Replay buffers<br/>[tensordict.py]")]
  node_samplers["Replay samplers<br/>[random.py]"]
  node_datasets["RL datasets"]
end

subgraph group_learning["Learning"]
  node_objectives["Objectives and losses<br/>[common.py]"]
  node_value["Value estimation<br/>[advantages.py]"]
  node_trainers["Training orchestration<br/>[trainers.py]"]
  node_models["Models and planners<br/>[models.py]"]
  node_llm_reward["LLM reward parsing<br/>[math.py]"]
end

subgraph group_operations["Runtime operations"]
  node_distributed["Distributed runtime<br/>[distributed.py]"]
  node_weight_sync["Weight synchronization"]
  node_logging["Logging<br/>[common.py]"]
  node_checkpoint["Checkpointing<br/>[_checkpoint.py]"]
end

node_user(("RL developer"))

node_user -->|"configures"| node_policies
node_user -->|"configures"| node_environments
node_policies -->|"writes actions"| node_environments
node_environments -->|"returns observations"| node_policies
node_environments -->|"wraps with"| node_transforms
node_environments -->|"exposes specs"| node_specs
node_policies -->|"writes actions"| node_tensordict
node_environments -->|"writes transitions"| node_tensordict
node_collectors -->|"collects trajectories"| node_environments
node_collectors -->|"batches data"| node_tensordict
node_collectors -->|"stores data"| node_replay
node_replay -.->|"samples data"| node_samplers
node_replay -->|"supplies batches"| node_objectives
node_tensordict -->|"feeds named keys"| node_objectives
node_objectives -.->|"uses estimates"| node_value
node_trainers -->|"orchestrates"| node_collectors
node_trainers -.->|"optimizes losses"| node_objectives
node_trainers -.->|"records metrics"| node_logging
node_trainers -.->|"saves state"| node_checkpoint
node_trainers -.->|"syncs weights"| node_weight_sync
node_distributed -.->|"scales workers"| node_collectors
node_models -.->|"builds policies"| node_policies
node_llm -.->|"scores responses"| node_llm_reward
node_datasets -.->|"provides offline data"| node_replay

click node_tensordict "https://github.com/pytorch/rl/tree/main/torchrl/data"
click node_specs "https://github.com/pytorch/rl/blob/main/torchrl/data/tensor_specs.py"
click node_environments "https://github.com/pytorch/rl/blob/main/torchrl/envs/common.py"
click node_transforms "https://github.com/pytorch/rl/blob/main/torchrl/envs/transforms/transforms.py"
click node_policies "https://github.com/pytorch/rl/blob/main/torchrl/modules/tensordict_module/actors.py"
click node_collectors "https://github.com/pytorch/rl/blob/main/torchrl/collectors/collectors.py"
click node_replay "https://github.com/pytorch/rl/blob/main/torchrl/data/replay_buffers/replay_buffers/tensordict.py"
click node_samplers "https://github.com/pytorch/rl/blob/main/torchrl/data/replay_buffers/samplers/random.py"
click node_datasets "https://github.com/pytorch/rl/tree/main/torchrl/data/datasets"
click node_objectives "https://github.com/pytorch/rl/blob/main/torchrl/objectives/common.py"
click node_value "https://github.com/pytorch/rl/blob/main/torchrl/objectives/value/advantages.py"
click node_trainers "https://github.com/pytorch/rl/blob/main/torchrl/trainers/trainers.py"
click node_models "https://github.com/pytorch/rl/blob/main/torchrl/modules/models/models.py"
click node_llm "https://github.com/pytorch/rl/blob/main/torchrl/envs/llm/envs.py"
click node_llm_reward "https://github.com/pytorch/rl/blob/main/torchrl/envs/llm/reward/math.py"
click node_distributed "https://github.com/pytorch/rl/blob/main/torchrl/_comm/distributed.py"
click node_weight_sync "https://github.com/pytorch/rl/blob/main/torchrl/weight_update/weight_sync_schemes.py"
click node_logging "https://github.com/pytorch/rl/blob/main/torchrl/record/loggers/common.py"
click node_checkpoint "https://github.com/pytorch/rl/blob/main/torchrl/checkpoint/_checkpoint.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_environments,node_transforms,node_policies,node_llm toneBlue
class node_tensordict,node_specs,node_collectors,node_replay,node_samplers,node_datasets toneAmber
class node_objectives,node_value,node_trainers,node_models,node_llm_reward toneMint
class node_distributed,node_weight_sync,node_logging,node_checkpoint toneRose
class node_user toneIndigo
```

## Components

### Interaction

- [Environments](https://github.com/pytorch/rl/blob/main/torchrl/envs/common.py): `torchrl/envs/common.py`
- [Environment transforms](https://github.com/pytorch/rl/blob/main/torchrl/envs/transforms/transforms.py): `torchrl/envs/transforms/transforms.py`
- [Policy modules](https://github.com/pytorch/rl/blob/main/torchrl/modules/tensordict_module/actors.py): `torchrl/modules/tensordict_module/actors.py`
- [LLM environments and policies](https://github.com/pytorch/rl/blob/main/torchrl/envs/llm/envs.py): `torchrl/envs/llm/envs.py`

### Data pipeline

- [TensorDict data](https://github.com/pytorch/rl/tree/main/torchrl/data): `torchrl/data`
- [Tensor specs](https://github.com/pytorch/rl/blob/main/torchrl/data/tensor_specs.py): `torchrl/data/tensor_specs.py`
- [Collectors](https://github.com/pytorch/rl/blob/main/torchrl/collectors/collectors.py): `torchrl/collectors/collectors.py`
- [Replay buffers](https://github.com/pytorch/rl/blob/main/torchrl/data/replay_buffers/replay_buffers/tensordict.py): `torchrl/data/replay_buffers/replay_buffers/tensordict.py`
- [Replay samplers](https://github.com/pytorch/rl/blob/main/torchrl/data/replay_buffers/samplers/random.py): `torchrl/data/replay_buffers/samplers/random.py`
- [RL datasets](https://github.com/pytorch/rl/tree/main/torchrl/data/datasets): `torchrl/data/datasets`

### Learning

- [Objectives and losses](https://github.com/pytorch/rl/blob/main/torchrl/objectives/common.py): `torchrl/objectives/common.py`
- [Value estimation](https://github.com/pytorch/rl/blob/main/torchrl/objectives/value/advantages.py): `torchrl/objectives/value/advantages.py`
- [Training orchestration](https://github.com/pytorch/rl/blob/main/torchrl/trainers/trainers.py): `torchrl/trainers/trainers.py`
- [Models and planners](https://github.com/pytorch/rl/blob/main/torchrl/modules/models/models.py): `torchrl/modules/models/models.py`
- [LLM reward parsing](https://github.com/pytorch/rl/blob/main/torchrl/envs/llm/reward/math.py): `torchrl/envs/llm/reward/math.py`

### Runtime operations

- [Distributed runtime](https://github.com/pytorch/rl/blob/main/torchrl/_comm/distributed.py): `torchrl/_comm/distributed.py`
- [Weight synchronization](https://github.com/pytorch/rl/blob/main/torchrl/weight_update/weight_sync_schemes.py): `torchrl/weight_update/weight_sync_schemes.py`
- [Logging](https://github.com/pytorch/rl/blob/main/torchrl/record/loggers/common.py): `torchrl/record/loggers/common.py`
- [Checkpointing](https://github.com/pytorch/rl/blob/main/torchrl/checkpoint/_checkpoint.py): `torchrl/checkpoint/_checkpoint.py`

### Other components

- **RL developer**

## Connections

- RL developer → Policy modules: configures
- RL developer → Environments: configures
- Policy modules → Environments: writes actions
- Environments → Policy modules: returns observations
- Environments → Environment transforms: wraps with (evidence: `torchrl/envs/common.py`)
- Environments → Tensor specs: exposes specs (evidence: `torchrl/envs/common.py`)
- Policy modules → TensorDict data: writes actions
- Environments → TensorDict data: writes transitions (evidence: `torchrl/envs/common.py`)
- Collectors → Environments: collects trajectories
- Collectors → TensorDict data: batches data
- Collectors → Replay buffers: stores data
- Replay buffers → Replay samplers: samples data (evidence: `torchrl/data/replay_buffers/samplers/random.py`)
- Replay buffers → Objectives and losses: supplies batches
- TensorDict data → Objectives and losses: feeds named keys
- Objectives and losses → Value estimation: uses estimates
- Training orchestration → Collectors: orchestrates
- Training orchestration → Objectives and losses: optimizes losses (evidence: `torchrl/trainers/trainers.py`)
- Training orchestration → Logging: records metrics
- Training orchestration → Checkpointing: saves state
- Training orchestration → Weight synchronization: syncs weights (evidence: `torchrl/trainers/trainers.py`)
- Distributed runtime → Collectors: scales workers
- Models and planners → Policy modules: builds policies
- LLM environments and policies → LLM reward parsing: scores responses
- RL datasets → Replay buffers: provides offline data
