# vwxyzjn/cleanrl architecture

> Architecture diagram of the GitHub repository vwxyzjn/cleanrl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vwxyzjn/cleanrl
- Repository: https://github.com/vwxyzjn/cleanrl

## Overview

CleanRL is a research-oriented deep-reinforcement-learning codebase: users run standalone algorithm scripts against an environment, train an agent, and inspect logged results or evaluate saved models. The algorithm portfolio spans PPO, value-based and actor-critic methods, Atari/Procgen, continuous-action and multi-agent tasks, and PyTorch/JAX or accelerated variants. Supporting branches provide evaluation, tuning and plotting, plus documented cloud experiment support. The README establishes several product capabilities; sampled source confirms selected training, evaluation, and environment interactions, so unsampled wiring is left implicit.

Main entry points: [`cleanrl`](https://github.com/vwxyzjn/cleanrl/tree/master/cleanrl) (PPO variants), [`cleanrl/ppo.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo.py) (PPO training), [`cleanrl_utils/docker_queue.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/docker_queue.py) (Cloud experiment queue), [`cleanrl_utils/evals`](https://github.com/vwxyzjn/cleanrl/tree/master/cleanrl_utils/evals) (Evaluators)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_training["Training"]
  node_ppo["PPO training<br/>[ppo.py]"]
  node_ppo_variants["PPO variants"]
  node_value_methods["Value methods"]
  node_actor_critic["Actor-critic methods"]
  node_specialized["Specialized methods"]
end

subgraph group_support["Training support"]
  node_buffers["Rollout/replay buffers<br/>[buffers.py]"]
  node_wrappers["Atari wrappers<br/>[atari_wrappers.py]"]
  node_evaluators["Evaluators"]
end

subgraph group_analysis["Experiment analysis"]
  node_tuner["Hyperparameter tuning<br/>[tuner.py]"]
  node_plots["Result plotting<br/>[plot.py]"]
  node_individual_plots["Run comparisons<br/>[plot_individual.py]"]
end

subgraph group_cloud["Cloud execution"]
  node_docker["Docker support<br/>[docker_build.py]"]
  node_cloud_queue["Cloud experiment queue<br/>[docker_queue.py]"]
end

node_researcher(("Researcher"))
node_environment["RL environment"]
node_tensorboard["TensorBoard"]
node_wandb["Weights &amp; Biases"]
node_model_hub["Hugging Face Hub"]
node_aws["AWS Batch"]

node_researcher -->|"runs script"| node_ppo
node_researcher -->|"selects variant"| node_ppo_variants
node_researcher -->|"runs algorithm"| node_value_methods
node_researcher -->|"runs algorithm"| node_actor_critic
node_researcher -->|"runs algorithm"| node_specialized
node_ppo -->|"interacts with"| node_environment
node_ppo_variants -->|"interacts with"| node_environment
node_value_methods -->|"interacts with"| node_environment
node_actor_critic -->|"interacts with"| node_environment
node_specialized -->|"interacts with"| node_environment
node_ppo_variants -->|"logs metrics"| node_tensorboard
node_researcher -.->|"tracks runs"| node_wandb
node_ppo -.->|"uses buffers"| node_buffers
node_ppo_variants -.->|"uses wrappers"| node_wrappers
node_researcher -->|"runs evaluation"| node_evaluators
node_evaluators -->|"evaluates in"| node_environment
node_model_hub -.->|"provides models"| node_evaluators
node_researcher -->|"configures trials"| node_tuner
node_tuner -.->|"launches trials"| node_ppo
node_researcher -->|"analyzes results"| node_plots
node_researcher -->|"compares runs"| node_individual_plots
node_researcher -->|"submits experiments"| node_cloud_queue
node_cloud_queue -.->|"dispatches jobs"| node_docker
node_cloud_queue -.->|"submits batches"| node_aws

click node_ppo "https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo.py"
click node_ppo_variants "https://github.com/vwxyzjn/cleanrl/tree/master/cleanrl"
click node_value_methods "https://github.com/vwxyzjn/cleanrl/tree/master/cleanrl"
click node_actor_critic "https://github.com/vwxyzjn/cleanrl/tree/master/cleanrl"
click node_specialized "https://github.com/vwxyzjn/cleanrl/tree/master/cleanrl"
click node_buffers "https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/buffers.py"
click node_wrappers "https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/atari_wrappers.py"
click node_evaluators "https://github.com/vwxyzjn/cleanrl/tree/master/cleanrl_utils/evals"
click node_tuner "https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/tuner.py"
click node_plots "https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/plot.py"
click node_individual_plots "https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/plot_individual.py"
click node_docker "https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/docker_build.py"
click node_cloud_queue "https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/docker_queue.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ppo,node_ppo_variants,node_value_methods,node_actor_critic,node_specialized toneBlue
class node_buffers,node_wrappers,node_evaluators toneAmber
class node_tuner,node_plots,node_individual_plots toneMint
class node_docker,node_cloud_queue toneRose
class node_researcher,node_environment,node_tensorboard,node_wandb,node_model_hub,node_aws toneIndigo
```

## Components

### Training

- [PPO training](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo.py): `cleanrl/ppo.py`
- [PPO variants](https://github.com/vwxyzjn/cleanrl/tree/master/cleanrl): `cleanrl`
- [Value methods](https://github.com/vwxyzjn/cleanrl/tree/master/cleanrl): `cleanrl`
- [Actor-critic methods](https://github.com/vwxyzjn/cleanrl/tree/master/cleanrl): `cleanrl`
- [Specialized methods](https://github.com/vwxyzjn/cleanrl/tree/master/cleanrl): `cleanrl`

### Training support

- [Rollout/replay buffers](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/buffers.py): `cleanrl_utils/buffers.py`
- [Atari wrappers](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/atari_wrappers.py): `cleanrl_utils/atari_wrappers.py`
- [Evaluators](https://github.com/vwxyzjn/cleanrl/tree/master/cleanrl_utils/evals): `cleanrl_utils/evals`

### Experiment analysis

- [Hyperparameter tuning](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/tuner.py): `cleanrl_utils/tuner.py`
- [Result plotting](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/plot.py): `cleanrl_utils/plot.py`
- [Run comparisons](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/plot_individual.py): `cleanrl_utils/plot_individual.py`

### Cloud execution

- [Docker support](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/docker_build.py): `cleanrl_utils/docker_build.py`
- [Cloud experiment queue](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl_utils/docker_queue.py): `cleanrl_utils/docker_queue.py`

### Other components

- **Researcher**
- **RL environment**
- **TensorBoard**
- **Weights & Biases**
- **Hugging Face Hub**
- **AWS Batch**

## Connections

- Researcher → PPO training: runs script
- Researcher → PPO variants: selects variant
- Researcher → Value methods: runs algorithm
- Researcher → Actor-critic methods: runs algorithm
- Researcher → Specialized methods: runs algorithm
- PPO training → RL environment: interacts with
- PPO variants → RL environment: interacts with
- Value methods → RL environment: interacts with
- Actor-critic methods → RL environment: interacts with
- Specialized methods → RL environment: interacts with
- PPO variants → TensorBoard: logs metrics
- Researcher → Weights & Biases: tracks runs
- PPO training → Rollout/replay buffers: uses buffers
- PPO variants → Atari wrappers: uses wrappers
- Researcher → Evaluators: runs evaluation
- Evaluators → RL environment: evaluates in
- Hugging Face Hub → Evaluators: provides models
- Researcher → Hyperparameter tuning: configures trials
- Hyperparameter tuning → PPO training: launches trials
- Researcher → Result plotting: analyzes results
- Researcher → Run comparisons: compares runs
- Researcher → Cloud experiment queue: submits experiments
- Cloud experiment queue → Docker support: dispatches jobs
- Cloud experiment queue → AWS Batch: submits batches