# openrl-lab/openrl architecture

> Architecture diagram of the GitHub repository openrl-lab/openrl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openrl-lab/openrl
- Repository: https://github.com/openrl-lab/openrl

## Overview

##### Project type and purpose
- This is a **full-stack reinforcement learning framework** written in Python (OpenRL).
- It provides:
- a **core library** of RL components (algorithms, networks, buffers, environments, drivers, self-play, etc.)
- a **command-line interface** (`openrl --mode train/test …`) for end users
- **example scripts** and **configs** for dozens of tasks (CartPole, Atari, multi-agent, NLP, etc.)
- **packaging and deployment** support (Conda, Docker, PyPI)
- **documentation**, **tests**, and **CI/CD pipelines**

##### File structure
- Top-level directories:
- `openrl/`: the **core library**, broken down into subpackages (algorithms, modules, envs, buffers, runners, selfplay, rewards, supports, utils, etc.)
- `examples/`: dozens of **task-specific scripts and YAML configs** showing how to use the library
- `scripts/`: **build/deploy/test pipelines** (Docker, Conda, PyPI, docs)
- `docker/`, `conda/`: **packaging recipes**
- `tests/`: **unit tests** covering core library, examples, CLI, algorithms, envs, etc.
- `README.md`, `docs/`, `.github/workflows/`: **documentation** and **CI**

##### README for additional insights
- Emphasizes a **unified interface** for single/multi-agent, offline RL, self-play, NLP, DeepSpeed, Arena evaluation, HuggingFace integration
- Lists supported algorithms (PPO, DQN, SAC, GAIL, MAT, VDN, BC, A2C, DDPG) and environments (Gymnasium, PettingZoo, Atari, SMAC, DM Control, etc.)
- Quick-start code shows the flow:
1. create env via `openrl.envs.common.make()`
2. create network via `openrl.modules.common.*Net`
3. create agent via `openrl.runners.common.*Agent`
4. call `agent.train()`

##### Detail
- **callback hooks** between Runner/Driver and Supports/Utils

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% API Layer
    subgraph "API Layer"
        CLI["CLI Layer"]:::core
        Config["Configuration Layer"]:::core
    end

    %% Core Engine
    subgraph "Core Engine"
        Runner["Runner/Agent Layer"]:::core
        Driver["Driver/Loop Layer"]:::core
        Buffer["Buffer Layer"]:::core
        Module["Module/Network Layer"]:::core
    end

    %% Environment Layer
    subgraph "Environment Layer"
        Env["Environment Layer"]:::core
    end

    %% Self-Play & Arena
    subgraph "Self-Play & Arena Layer"
        SelfPlay["Self-Play Layer"]:::core
        Arena["Arena Layer"]:::core
    end

    %% Reward & Dataset
    subgraph "Reward & Dataset Layer"
        Reward["Reward Layer"]:::core
        Dataset["Dataset Layer"]:::core
    end

    %% Supports & Utils
    subgraph "Support & Utilities Layer"
        Supports["Supports Layer"]:::core
        Utils["Utilities Layer"]:::core
    end

    %% Examples & Tests
    subgraph "Examples & Tests"
        Examples["Examples"]:::example
        Tests["Tests"]:::example
    end

    %% Packaging & Deployment
    subgraph "Packaging & Deployment"
        Packaging["Packaging & CI"]:::pack
    end

    %% External Systems
    PyTorch["PyTorch"]:::external
    Gym["Gymnasium/PettingZoo/SMAC/DMControl"]:::external
    DeepSpeed["DeepSpeed"]:::external
    HuggingFace["HuggingFace"]:::external
    WandB["WandB"]:::external
    TensorBoard["TensorBoardX"]:::external
    DockerExt["Docker"]:::external
    CondaExt["Conda"]:::external
    GitHubCI["GitHub Actions"]:::external

    %% Connections: Control Flow and Data Flow
    CLI -->|"dispatch train/test"| Runner
    CLI -->|"load configs"| Config
    Runner -->|"create driver"| Driver
    Driver -->|"env.step()"| Env
    Env -->|"obs"| Driver
    Driver -->|"buffer.push()"| Buffer
    Runner -->|"train_step()"| Module
    Module -->|"forward()/backward()"| PyTorch
    Module -->|"sample action"| Driver
    Runner -->|"load dataset"| Dataset
    Runner -->|"compute reward"| Reward
    Runner -->|"self-play loop"| SelfPlay
    SelfPlay -->|"arena matches"| Arena
    Runner -->|"callbacks/hooks"| Supports
    Driver -->|"event hooks"| Utils
    Examples -->|"invoke CLI/API"| CLI
    Tests -->|"test layers"| Runner
    Packaging -->|"build/push"| DockerExt
    Packaging -->|"build/publish"| CondaExt
    Packaging -->|"CI pipelines"| GitHubCI

    %% External integrations
    PyTorch ---|uses| DeepSpeed
    Runner ---|log metrics| WandB
    Runner ---|log metrics| TensorBoard
    Env ---|provides| Gym

    %% Click Events
    click CLI "https://github.com/openrl-lab/openrl/blob/main/openrl/cli/cli.py"
    click Config "https://github.com/openrl-lab/openrl/blob/main/openrl/configs/config.py"
    click Runner "https://github.com/openrl-lab/openrl/blob/main/openrl/runners/common/ppo_agent.py"
    click Driver "https://github.com/openrl-lab/openrl/blob/main/openrl/drivers/onpolicy_driver.py"
    click Buffer "https://github.com/openrl-lab/openrl/blob/main/openrl/buffers/normal_buffer.py"
    click Module "https://github.com/openrl-lab/openrl/blob/main/openrl/modules/ppo_module.py"
    click Env "https://github.com/openrl-lab/openrl/blob/main/openrl/envs/common/registration.py"
    click SelfPlay "https://github.com/openrl-lab/openrl/blob/main/openrl/selfplay/selfplay_api/selfplay_api.py"
    click Arena "https://github.com/openrl-lab/openrl/blob/main/openrl/arena/base_arena.py"
    click Reward "https://github.com/openrl-lab/openrl/blob/main/openrl/rewards/base_reward.py"
    click Dataset "https://github.com/openrl-lab/openrl/blob/main/openrl/datasets/expert_dataset.py"
    click Supports "https://github.com/openrl-lab/openrl/blob/main/openrl/supports/opengpu/gpu_info.py"
    click Utils "https://github.com/openrl-lab/openrl/blob/main/openrl/utils/logger.py"
    click Examples "https://github.com/openrl-lab/openrl/blob/main/examples/cartpole/train_ppo.py"
    click Tests "https://github.com/openrl-lab/openrl/tree/main/tests/test_algorithm/"
    click Packaging "https://github.com/openrl-lab/openrl/tree/main/docker/Dockerfile"

    %% Styles
    classDef core fill:#cfe2f3,stroke:#2a5d84,color:#000;
    classDef external fill:#d5e8d4,stroke:#4b8b3b,color:#000;
    classDef example fill:#fff2cc,stroke:#a67c00,color:#000;
    classDef pack fill:#fbd5b5,stroke:#b85403,color:#000;
```
