# dlr-rm/stable-baselines3 architecture

> Architecture diagram of the GitHub repository dlr-rm/stable-baselines3, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dlr-rm/stable-baselines3
- Repository: https://github.com/dlr-rm/stable-baselines3

## Overview

Stable-Baselines3 is a PyTorch reinforcement-learning library. A user selects an algorithm and policy, connects an environment, trains through repeated environment interaction and learning updates, then uses the trained model to predict actions. On-policy algorithms collect rollouts; off-policy algorithms store transitions and learn from replay. Shared policies, vectorized environments, buffers, callbacks, logging, and evaluation support these paths. The README documents custom environments and policies, Dict observations, callbacks, and TensorBoard support; the sampled source confirms selected policy, buffer, and vector-environment details, so uncertain internal wiring is omitted.

Main entry points: [`stable_baselines3/common/off_policy_algorithm.py`](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/off_policy_algorithm.py) (Off-policy loop), [`stable_baselines3/common/base_class.py`](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/base_class.py) (Model interface), [`stable_baselines3/common/on_policy_algorithm.py`](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/on_policy_algorithm.py) (On-policy loop), [`stable_baselines3/sac/sac.py`](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/sac/sac.py) (SAC)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_algorithms["RL algorithms"]
  node_a2c["A2C<br/>[a2c.py]"]
  node_ppo["PPO<br/>[ppo.py]"]
  node_dqn["DQN<br/>[dqn.py]"]
  node_ddpg["DDPG<br/>[ddpg.py]"]
  node_sac["SAC<br/>[sac.py]"]
  node_td3["TD3<br/>[td3.py]"]
end

subgraph group_runtime["Training runtime"]
  node_base["Model interface<br/>[base_class.py]"]
  node_onpolicy["On-policy loop"]
  node_offpolicy["Off-policy loop"]
end

subgraph group_learning["Policies and learning"]
  node_policy["Shared policies<br/>[policies.py]"]
  node_algo_policies["Algorithm policies<br/>[policies.py]"]
  node_rollout[("Rollout buffer<br/>[buffers.py]")]
  node_replay[("Replay buffer<br/>[buffers.py]")]
  node_her[("HER replay")]
end

subgraph group_environment["Environment tools"]
  node_vecenv["Vector environments<br/>[base_vec_env.py]"]
  node_wrappers["Environment wrappers<br/>[atari_wrappers.py]"]
  node_customenvs["Example environments"]
end

subgraph group_observability["Monitoring and evaluation"]
  node_callbacks["Callbacks<br/>[callbacks.py]"]
  node_logger["Logger<br/>[logger.py]"]
  node_evaluation["Evaluation<br/>[evaluation.py]"]
end

node_user(("RL user"))
node_gym["Gymnasium environment"]
node_tensorboard["TensorBoard"]

node_user -->|"selects and trains"| node_a2c
node_user -->|"selects and trains"| node_ppo
node_user -->|"selects and trains"| node_dqn
node_user -->|"selects and trains"| node_ddpg
node_user -->|"selects and trains"| node_sac
node_user -->|"selects and trains"| node_td3
node_a2c -->|"uses loop"| node_onpolicy
node_ppo -->|"uses loop"| node_onpolicy
node_dqn -->|"uses loop"| node_offpolicy
node_ddpg -->|"uses loop"| node_offpolicy
node_sac -->|"uses loop"| node_offpolicy
node_td3 -->|"uses loop"| node_offpolicy
node_a2c -->|"extends interface"| node_base
node_ppo -->|"extends interface"| node_base
node_dqn -->|"extends interface"| node_base
node_ddpg -->|"extends interface"| node_base
node_sac -->|"extends interface"| node_base
node_td3 -->|"extends interface"| node_base
node_onpolicy -->|"steps environments"| node_vecenv
node_offpolicy -->|"steps environments"| node_vecenv
node_vecenv -->|"steps"| node_gym
node_wrappers -->|"wraps"| node_gym
node_customenvs -->|"implements environments"| node_gym
node_onpolicy -->|"updates policy"| node_policy
node_offpolicy -->|"updates policy"| node_policy
node_sac -->|"selects policy"| node_algo_policies
node_td3 -->|"selects policy"| node_algo_policies
node_onpolicy -->|"stores rollouts"| node_rollout
node_offpolicy -->|"stores transitions"| node_replay
node_her -->|"extends replay"| node_replay
node_offpolicy -.->|"configures buffer"| node_her
node_onpolicy -->|"dispatches callbacks"| node_callbacks
node_offpolicy -->|"dispatches callbacks"| node_callbacks
node_base -->|"records metrics"| node_logger
node_user -->|"requests evaluation"| node_evaluation
node_evaluation -->|"runs episodes"| node_vecenv
node_logger -.->|"exports metrics"| node_tensorboard

click node_a2c "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/a2c/a2c.py"
click node_ppo "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/ppo/ppo.py"
click node_dqn "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/dqn/dqn.py"
click node_ddpg "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/ddpg/ddpg.py"
click node_sac "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/sac/sac.py"
click node_td3 "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/td3/td3.py"
click node_base "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/base_class.py"
click node_onpolicy "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/on_policy_algorithm.py"
click node_offpolicy "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/off_policy_algorithm.py"
click node_vecenv "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/vec_env/base_vec_env.py"
click node_wrappers "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/atari_wrappers.py"
click node_customenvs "https://github.com/dlr-rm/stable-baselines3/tree/master/stable_baselines3/common/envs"
click node_policy "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/policies.py"
click node_algo_policies "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/sac/policies.py"
click node_rollout "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/buffers.py"
click node_replay "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/buffers.py"
click node_her "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/her/her_replay_buffer.py"
click node_callbacks "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/callbacks.py"
click node_logger "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/logger.py"
click node_evaluation "https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/evaluation.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_a2c,node_ppo,node_dqn,node_ddpg,node_sac,node_td3,node_user toneBlue
class node_base,node_onpolicy,node_offpolicy toneAmber
class node_policy,node_algo_policies,node_rollout,node_replay,node_her toneMint
class node_vecenv,node_wrappers,node_customenvs toneRose
class node_callbacks,node_logger,node_evaluation,node_gym,node_tensorboard toneIndigo
```

## Components

### RL algorithms

- [A2C](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/a2c/a2c.py): `stable_baselines3/a2c/a2c.py`
- [PPO](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/ppo/ppo.py): `stable_baselines3/ppo/ppo.py`
- [DQN](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/dqn/dqn.py): `stable_baselines3/dqn/dqn.py`
- [DDPG](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/ddpg/ddpg.py): `stable_baselines3/ddpg/ddpg.py`
- [SAC](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/sac/sac.py): `stable_baselines3/sac/sac.py`
- [TD3](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/td3/td3.py): `stable_baselines3/td3/td3.py`

### Training runtime

- [Model interface](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/base_class.py): `stable_baselines3/common/base_class.py`
- [On-policy loop](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/on_policy_algorithm.py): `stable_baselines3/common/on_policy_algorithm.py`
- [Off-policy loop](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/off_policy_algorithm.py): `stable_baselines3/common/off_policy_algorithm.py`

### Policies and learning

- [Shared policies](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/policies.py): `stable_baselines3/common/policies.py`
- [Algorithm policies](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/sac/policies.py): `stable_baselines3/sac/policies.py`
- [Rollout buffer](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/buffers.py): `stable_baselines3/common/buffers.py`
- [Replay buffer](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/buffers.py): `stable_baselines3/common/buffers.py`
- [HER replay](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/her/her_replay_buffer.py): `stable_baselines3/her/her_replay_buffer.py`

### Environment tools

- [Vector environments](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/vec_env/base_vec_env.py): `stable_baselines3/common/vec_env/base_vec_env.py`
- [Environment wrappers](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/atari_wrappers.py): `stable_baselines3/common/atari_wrappers.py`
- [Example environments](https://github.com/dlr-rm/stable-baselines3/tree/master/stable_baselines3/common/envs): `stable_baselines3/common/envs`

### Monitoring and evaluation

- [Callbacks](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/callbacks.py): `stable_baselines3/common/callbacks.py`
- [Logger](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/logger.py): `stable_baselines3/common/logger.py`
- [Evaluation](https://github.com/dlr-rm/stable-baselines3/blob/master/stable_baselines3/common/evaluation.py): `stable_baselines3/common/evaluation.py`

### Other components

- **RL user**
- **Gymnasium environment**
- **TensorBoard**

## Connections

- RL user → A2C: selects and trains
- RL user → PPO: selects and trains
- RL user → DQN: selects and trains
- RL user → DDPG: selects and trains
- RL user → SAC: selects and trains
- RL user → TD3: selects and trains
- A2C → On-policy loop: uses loop
- PPO → On-policy loop: uses loop
- DQN → Off-policy loop: uses loop
- DDPG → Off-policy loop: uses loop
- SAC → Off-policy loop: uses loop
- TD3 → Off-policy loop: uses loop
- A2C → Model interface: extends interface
- PPO → Model interface: extends interface
- DQN → Model interface: extends interface
- DDPG → Model interface: extends interface
- SAC → Model interface: extends interface
- TD3 → Model interface: extends interface
- On-policy loop → Vector environments: steps environments
- Off-policy loop → Vector environments: steps environments
- Vector environments → Gymnasium environment: steps
- Environment wrappers → Gymnasium environment: wraps
- Example environments → Gymnasium environment: implements environments
- On-policy loop → Shared policies: updates policy
- Off-policy loop → Shared policies: updates policy
- SAC → Algorithm policies: selects policy
- TD3 → Algorithm policies: selects policy
- On-policy loop → Rollout buffer: stores rollouts
- Off-policy loop → Replay buffer: stores transitions
- HER replay → Replay buffer: extends replay
- Off-policy loop → HER replay: configures buffer
- On-policy loop → Callbacks: dispatches callbacks
- Off-policy loop → Callbacks: dispatches callbacks
- Model interface → Logger: records metrics
- RL user → Evaluation: requests evaluation
- Evaluation → Vector environments: runs episodes
- Logger → TensorBoard: exports metrics