# quantumiracle/popular-rl-algorithms architecture

> Architecture diagram of the GitHub repository quantumiracle/popular-rl-algorithms, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/quantumiracle/popular-rl-algorithms
- Repository: https://github.com/quantumiracle/popular-rl-algorithms

## Overview

This repository is a research collection of model-free reinforcement-learning implementations, not a unified application or library. Its common user workflow is to run an algorithm script against an environment, collect experience, update a policy or value model, and evaluate behavior. The README documents additional algorithm families whose individual wiring was not sampled; those are shown as separate capability branches rather than an invented shared pipeline. Source excerpts confirm SAC components, shared network and buffer utilities, and the CEM rollout loop.

Main entry points: [`sac_v2.py`](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/sac_v2.py) (SAC v2), [`cem/cem.py`](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/cem/cem.py) (Cross-entropy method), [`sdt_ppo_gae_discrete.py`](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/sdt_ppo_gae_discrete.py) (Tree-based PPO), [`cem/_policies.py`](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/cem/_policies.py) (Linear policies)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_offpolicy["Off-policy RL"]
  node_sac["SAC v2<br/>[sac_v2.py]"]
  node_sacdisc["Discrete SAC<br/>[sac_discrete.py]"]
  node_ddpg["DDPG<br/>[ddpg.py]"]
  node_td3["TD3<br/>[td3.py]"]
  node_dqn["DQN<br/>[dqn.py]"]
  node_recurrent["Recurrent policies<br/>[rdpg.py]"]
end

subgraph group_onpolicy["On-policy RL"]
  node_ppo["PPO variants<br/>[ppo_continuous.py]"]
  node_ac["Actor-Critic<br/>[ac.py]"]
end

subgraph group_specialized["Specialized methods"]
  node_qmix["QMIX<br/>[qmix.py]"]
  node_pmoe["Mixture policies<br/>[pmoe_sac.py]"]
  node_sdtppo["Tree-based PPO"]
  node_sdt["Soft decision tree<br/>[SDT.py]"]
  node_cem["Cross-entropy method<br/>[cem.py]"]
  node_qtopt["QT-Opt<br/>[qt_opt_v3.py]"]
  node_cem_policy["Linear policies<br/>[_policies.py]"]
end

subgraph group_shared["Shared components"]
  node_buffers[("Replay buffers<br/>[buffers.py]")]
  node_networks["Policy/value networks<br/>[policy_networks.py]"]
  node_initialize["Network initialization<br/>[initialize.py]"]
end

node_user(("Experimenter"))
node_gym["Gym environments"]
node_reacher["Reacher environment<br/>[reacher.py]"]

node_user -->|"runs training"| node_sac
node_user -->|"runs training"| node_ppo
node_user -->|"runs experiment"| node_cem
node_sac -->|"steps environment"| node_gym
node_sac -->|"stores and samples"| node_buffers
node_sac -.->|"uses networks"| node_networks
node_networks -->|"imports initialization"| node_initialize
node_user -->|"selects environment"| node_reacher
node_cem -->|"evaluates rollouts"| node_gym
node_cem -->|"creates policy"| node_cem_policy
node_cem_policy -->|"selects actions"| node_gym
node_user -->|"runs algorithm"| node_sacdisc
node_user -->|"runs algorithm"| node_ddpg
node_user -->|"runs algorithm"| node_td3
node_user -->|"runs algorithm"| node_dqn
node_user -->|"runs algorithm"| node_recurrent
node_user -->|"runs algorithm"| node_ac
node_user -->|"runs algorithm"| node_qmix
node_user -->|"runs algorithm"| node_pmoe
node_user -->|"runs algorithm"| node_sdtppo
node_sdtppo -.->|"uses tree model"| node_sdt
node_user -->|"runs algorithm"| node_qtopt

click node_reacher "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/reacher.py"
click node_sac "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/sac_v2.py"
click node_sacdisc "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/sac_discrete.py"
click node_ddpg "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/ddpg.py"
click node_td3 "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/td3.py"
click node_dqn "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/dqn.py"
click node_recurrent "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/rdpg.py"
click node_ppo "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/ppo_continuous.py"
click node_ac "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/ac.py"
click node_qmix "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/qmix.py"
click node_pmoe "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/pmoe_sac.py"
click node_sdtppo "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/sdt_ppo_gae_discrete.py"
click node_sdt "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/SDT/SDT.py"
click node_cem "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/cem/cem.py"
click node_qtopt "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/qt_opt_v3.py"
click node_buffers "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/common/buffers.py"
click node_networks "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/common/policy_networks.py"
click node_initialize "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/common/initialize.py"
click node_cem_policy "https://github.com/quantumiracle/popular-rl-algorithms/blob/master/cem/_policies.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sac,node_sacdisc,node_ddpg,node_td3,node_dqn,node_recurrent toneBlue
class node_ppo,node_ac toneAmber
class node_qmix,node_pmoe,node_sdtppo,node_sdt,node_cem,node_qtopt,node_cem_policy toneMint
class node_buffers,node_networks,node_initialize toneRose
class node_user,node_gym toneIndigo
class node_reacher toneTeal
```

## Components

### Off-policy RL

- [SAC v2](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/sac_v2.py): `sac_v2.py`
- [Discrete SAC](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/sac_discrete.py): `sac_discrete.py`
- [DDPG](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/ddpg.py): `ddpg.py`
- [TD3](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/td3.py): `td3.py`
- [DQN](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/dqn.py): `dqn.py`
- [Recurrent policies](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/rdpg.py): `rdpg.py`

### On-policy RL

- [PPO variants](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/ppo_continuous.py): `ppo_continuous.py`
- [Actor-Critic](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/ac.py): `ac.py`

### Specialized methods

- [QMIX](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/qmix.py): `qmix.py`
- [Mixture policies](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/pmoe_sac.py): `pmoe_sac.py`
- [Tree-based PPO](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/sdt_ppo_gae_discrete.py): `sdt_ppo_gae_discrete.py`
- [Soft decision tree](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/SDT/SDT.py): `SDT/SDT.py`
- [Cross-entropy method](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/cem/cem.py): `cem/cem.py`
- [QT-Opt](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/qt_opt_v3.py): `qt_opt_v3.py`
- [Linear policies](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/cem/_policies.py): `cem/_policies.py`

### Shared components

- [Replay buffers](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/common/buffers.py): `common/buffers.py`
- [Policy/value networks](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/common/policy_networks.py): `common/policy_networks.py`
- [Network initialization](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/common/initialize.py): `common/initialize.py`

### Other components

- **Experimenter**
- **Gym environments**
- [Reacher environment](https://github.com/quantumiracle/popular-rl-algorithms/blob/master/reacher.py): `reacher.py`

## Connections

- Experimenter → SAC v2: runs training
- Experimenter → PPO variants: runs training
- Experimenter → Cross-entropy method: runs experiment
- SAC v2 → Gym environments: steps environment
- SAC v2 → Replay buffers: stores and samples
- SAC v2 → Policy/value networks: uses networks
- Policy/value networks → Network initialization: imports initialization
- Experimenter → Reacher environment: selects environment
- Cross-entropy method → Gym environments: evaluates rollouts
- Cross-entropy method → Linear policies: creates policy
- Linear policies → Gym environments: selects actions
- Experimenter → Discrete SAC: runs algorithm
- Experimenter → DDPG: runs algorithm
- Experimenter → TD3: runs algorithm
- Experimenter → DQN: runs algorithm
- Experimenter → Recurrent policies: runs algorithm
- Experimenter → Actor-Critic: runs algorithm
- Experimenter → QMIX: runs algorithm
- Experimenter → Mixture policies: runs algorithm
- Experimenter → Tree-based PPO: runs algorithm
- Tree-based PPO → Soft decision tree: uses tree model
- Experimenter → QT-Opt: runs algorithm