How vwxyzjn/cleanrl works
CleanRL is a research-oriented deep-reinforcement-learning codebase: users run standalone algorithm scripts against an environment, train an agent, and inspect logged results or evaluate saved models. The algorithm portfolio spans PPO, value-based and actor-critic methods, Atari/Procgen, continuous-action and multi-agent tasks, and PyTorch/JAX or accelerated variants. Supporting branches provide evaluation, tuning and plotting, plus documented cloud experiment support. The README establishes several product capabilities; sampled source confirms selected training, evaluation, and environment interactions, so unsampled wiring is left implicit.
Main entry points: cleanrl PPO variants · ppo.py PPO training · docker_queue.py Cloud experiment queue · evals Evaluators
Components 19
Training
- PPO training
cleanrl/ppo.py - PPO variants
cleanrl - Value methods
cleanrl - Actor-critic methods
cleanrl - Specialized methods
cleanrl
Training support
- Rollout/replay buffers
cleanrl_utils/buffers.py - Atari wrappers
cleanrl_utils/atari_wrappers.py - Evaluators
cleanrl_utils/evals
Experiment analysis
- Hyperparameter tuning
cleanrl_utils/tuner.py - Result plotting
cleanrl_utils/plot.py - Run comparisons
cleanrl_utils/plot_individual.py
Cloud execution
- Docker support
cleanrl_utils/docker_build.py - Cloud experiment queue
cleanrl_utils/docker_queue.py
Other components
- Researcher
- RL environment
- TensorBoard
- Weights & Biases
- Hugging Face Hub
- AWS Batch
Connections 24
- Researcher → PPO training: runs script
- Researcher → PPO variants: selects variant
- Researcher → Value methods: runs algorithm
- Researcher → Actor-critic methods: runs algorithm
- Researcher → Specialized methods: runs algorithm
- PPO training → RL environment: interacts with
- PPO variants → RL environment: interacts with
- Value methods → RL environment: interacts with
- Actor-critic methods → RL environment: interacts with
- Specialized methods → RL environment: interacts with
- PPO variants → TensorBoard: logs metrics
- Researcher → Weights & Biases: tracks runs
- PPO training → Rollout/replay buffers: uses buffers
- PPO variants → Atari wrappers: uses wrappers
- Researcher → Evaluators: runs evaluation
- Evaluators → RL environment: evaluates in
- Hugging Face Hub → Evaluators: provides models
- Researcher → Hyperparameter tuning: configures trials
- Hyperparameter tuning → PPO training: launches trials
- Researcher → Result plotting: analyzes results
- Researcher → Run comparisons: compares runs
- Researcher → Cloud experiment queue: submits experiments
- Cloud experiment queue → Docker support: dispatches jobs
- Cloud experiment queue → AWS Batch: submits batches