How pytorch/rl works
TorchRL is a composable, PyTorch-native reinforcement-learning toolkit organized around TensorDict data. The principal workflow is policy–environment interaction, collection of trajectories, optional replay storage and sampling, then objective computation and optimization. The README also documents broad environment and transform support, distributed and recurrent workflows, model-based and multi-agent learning, and LLM components. Edges below follow the documented mental model; sampled source excerpts are incomplete, so relationships not established by the README or visible references are omitted.
Main entry points: common.py Environments · trainers.py Training orchestration · collectors.py Collectors · actors.py Policy modules
Components 20
Interaction
- Environments
torchrl/envs/common.py - Environment transforms
torchrl/envs/transforms/transforms.py - Policy modules
torchrl/modules/tensordict_module/actors.py - LLM environments and policies
torchrl/envs/llm/envs.py
Data pipeline
- TensorDict data
torchrl/data - Tensor specs
torchrl/data/tensor_specs.py - Collectors
torchrl/collectors/collectors.py - Replay buffers
torchrl/data/replay_buffers/replay_buffers/tensordict.py - Replay samplers
torchrl/data/replay_buffers/samplers/random.py - RL datasets
torchrl/data/datasets
Learning
- Objectives and losses
torchrl/objectives/common.py - Value estimation
torchrl/objectives/value/advantages.py - Training orchestration
torchrl/trainers/trainers.py - Models and planners
torchrl/modules/models/models.py - LLM reward parsing
torchrl/envs/llm/reward/math.py
Runtime operations
- Distributed runtime
torchrl/_comm/distributed.py - Weight synchronization
torchrl/weight_update/weight_sync_schemes.py - Logging
torchrl/record/loggers/common.py - Checkpointing
torchrl/checkpoint/_checkpoint.py
Other components
- RL developer
Connections 24
- RL developer → Policy modules: configures
- RL developer → Environments: configures
- Policy modules → Environments: writes actions
- Environments → Policy modules: returns observations
- Environments → Environment transforms: wraps with
torchrl/envs/common.py - Environments → Tensor specs: exposes specs
torchrl/envs/common.py - Policy modules → TensorDict data: writes actions
- Environments → TensorDict data: writes transitions
torchrl/envs/common.py - Collectors → Environments: collects trajectories
- Collectors → TensorDict data: batches data
- Collectors → Replay buffers: stores data
- Replay buffers → Replay samplers: samples data
torchrl/data/replay_buffers/samplers/random.py - Replay buffers → Objectives and losses: supplies batches
- TensorDict data → Objectives and losses: feeds named keys
- Objectives and losses → Value estimation: uses estimates
- Training orchestration → Collectors: orchestrates
- Training orchestration → Objectives and losses: optimizes losses
torchrl/trainers/trainers.py - Training orchestration → Logging: records metrics
- Training orchestration → Checkpointing: saves state
- Training orchestration → Weight synchronization: syncs weights
torchrl/trainers/trainers.py - Distributed runtime → Collectors: scales workers
- Models and planners → Policy modules: builds policies
- LLM environments and policies → LLM reward parsing: scores responses
- RL datasets → Replay buffers: provides offline data