# thu-ml/tianshou architecture

> Architecture diagram of the GitHub repository thu-ml/tianshou, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thu-ml/tianshou
- Repository: https://github.com/thu-ml/tianshou

## Overview

Tianshou is a PyTorch reinforcement-learning library with procedural and high-level APIs. The principal workflow configures an algorithm and environment, collects interaction data, stores experience, trains the algorithm, and reports results. Its algorithm scope includes model-free, imitation, model-based, and multi-agent RL; it also supports vectorized environments, prioritized and hindsight replay, and experiment evaluation. The graph combines README-documented capabilities with sampled trainer, collector, environment, and experiment code; links for unsampled subsystem internals are limited to documented workflow relationships.

Main entry points: [`tianshou/algorithm/algorithm_base.py`](https://github.com/thu-ml/tianshou/blob/master/tianshou/algorithm/algorithm_base.py) (Algorithm interface), [`tianshou/data/collector.py`](https://github.com/thu-ml/tianshou/blob/master/tianshou/data/collector.py) (Experience collection), [`tianshou/trainer.py`](https://github.com/thu-ml/tianshou/blob/master/tianshou/trainer.py) (Training loop), [`tianshou/highlevel/experiment.py`](https://github.com/thu-ml/tianshou/blob/master/tianshou/highlevel/experiment.py) (Experiment setup)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Experiment APIs"]
  node_highlevel["Experiment setup<br/>[experiment.py]"]
end

subgraph group_runtime["Training Runtime"]
  node_trainer["Training loop<br/>[trainer.py]"]
  node_collector["Experience collection<br/>[collector.py]"]
end

subgraph group_learning["Learning Algorithms"]
  node_algorithm_base["Algorithm interface<br/>[algorithm_base.py]"]
  node_modelfree["Model-free RL"]
  node_imitation["Imitation RL"]
  node_modelbased["Model-based RL"]
  node_multiagent["Multi-agent RL"]
  node_networks["Actor and critic nets"]
end

subgraph group_environment["Environment and Data"]
  node_vector_env["Vector environments<br/>[venvs.py]"]
  node_workers["Environment workers<br/>[worker_base.py]"]
  node_gym["Gymnasium environments"]
  node_replay[("Replay buffers")]
  node_batch["Batch data<br/>[batch.py]"]
end

subgraph group_results["Results"]
  node_logger["Experiment logging<br/>[logger.py]"]
  node_evaluation["RL evaluation"]
  node_logger_backends["TensorBoard and W&amp;B"]
end

node_user(("Researcher or app"))

node_user -->|"configures experiment"| node_highlevel
node_user -->|"uses algorithm API"| node_algorithm_base
node_highlevel -->|"runs training"| node_trainer
node_highlevel -->|"creates environments"| node_vector_env
node_trainer -->|"collects experience"| node_collector
node_trainer -->|"requests updates"| node_algorithm_base
node_algorithm_base -->|"implements"| node_modelfree
node_algorithm_base -->|"implements"| node_imitation
node_algorithm_base -->|"implements"| node_modelbased
node_algorithm_base -->|"implements"| node_multiagent
node_modelfree -->|"uses networks"| node_networks
node_collector -->|"gets actions"| node_algorithm_base
node_collector -->|"steps environments"| node_vector_env
node_vector_env -->|"dispatches work"| node_workers
node_workers -->|"interacts with"| node_gym
node_collector -->|"stores experience"| node_replay
node_algorithm_base -->|"samples experience"| node_replay
node_collector -->|"produces batches"| node_batch
node_trainer -->|"reports metrics"| node_logger
node_logger -->|"writes logs"| node_logger_backends
node_evaluation -->|"reads experiment data"| node_logger

click node_highlevel "https://github.com/thu-ml/tianshou/blob/master/tianshou/highlevel/experiment.py"
click node_algorithm_base "https://github.com/thu-ml/tianshou/blob/master/tianshou/algorithm/algorithm_base.py"
click node_modelfree "https://github.com/thu-ml/tianshou/tree/master/tianshou/algorithm/modelfree"
click node_imitation "https://github.com/thu-ml/tianshou/tree/master/tianshou/algorithm/imitation"
click node_modelbased "https://github.com/thu-ml/tianshou/tree/master/tianshou/algorithm/modelbased"
click node_multiagent "https://github.com/thu-ml/tianshou/tree/master/tianshou/algorithm/multiagent"
click node_networks "https://github.com/thu-ml/tianshou/tree/master/tianshou/utils/net"
click node_trainer "https://github.com/thu-ml/tianshou/blob/master/tianshou/trainer.py"
click node_collector "https://github.com/thu-ml/tianshou/blob/master/tianshou/data/collector.py"
click node_vector_env "https://github.com/thu-ml/tianshou/blob/master/tianshou/env/venvs.py"
click node_workers "https://github.com/thu-ml/tianshou/blob/master/tianshou/env/worker/worker_base.py"
click node_replay "https://github.com/thu-ml/tianshou/tree/master/tianshou/data/buffer"
click node_batch "https://github.com/thu-ml/tianshou/blob/master/tianshou/data/batch.py"
click node_logger "https://github.com/thu-ml/tianshou/blob/master/tianshou/highlevel/logger.py"
click node_evaluation "https://github.com/thu-ml/tianshou/blob/master/tianshou/evaluation/rliable_evaluation.py"
click node_logger_backends "https://github.com/thu-ml/tianshou/tree/master/tianshou/utils/logger"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_highlevel toneBlue
class node_trainer,node_collector toneAmber
class node_algorithm_base,node_modelfree,node_imitation,node_modelbased,node_multiagent,node_networks toneMint
class node_vector_env,node_workers,node_gym,node_replay,node_batch toneRose
class node_logger,node_evaluation,node_logger_backends,node_user toneIndigo
```

## Components

### Experiment APIs

- [Experiment setup](https://github.com/thu-ml/tianshou/blob/master/tianshou/highlevel/experiment.py): `tianshou/highlevel/experiment.py`

### Training Runtime

- [Training loop](https://github.com/thu-ml/tianshou/blob/master/tianshou/trainer.py): `tianshou/trainer.py`
- [Experience collection](https://github.com/thu-ml/tianshou/blob/master/tianshou/data/collector.py): `tianshou/data/collector.py`

### Learning Algorithms

- [Algorithm interface](https://github.com/thu-ml/tianshou/blob/master/tianshou/algorithm/algorithm_base.py): `tianshou/algorithm/algorithm_base.py`
- [Model-free RL](https://github.com/thu-ml/tianshou/tree/master/tianshou/algorithm/modelfree): `tianshou/algorithm/modelfree`
- [Imitation RL](https://github.com/thu-ml/tianshou/tree/master/tianshou/algorithm/imitation): `tianshou/algorithm/imitation`
- [Model-based RL](https://github.com/thu-ml/tianshou/tree/master/tianshou/algorithm/modelbased): `tianshou/algorithm/modelbased`
- [Multi-agent RL](https://github.com/thu-ml/tianshou/tree/master/tianshou/algorithm/multiagent): `tianshou/algorithm/multiagent`
- [Actor and critic nets](https://github.com/thu-ml/tianshou/tree/master/tianshou/utils/net): `tianshou/utils/net`

### Environment and Data

- [Vector environments](https://github.com/thu-ml/tianshou/blob/master/tianshou/env/venvs.py): `tianshou/env/venvs.py`
- [Environment workers](https://github.com/thu-ml/tianshou/blob/master/tianshou/env/worker/worker_base.py): `tianshou/env/worker/worker_base.py`
- **Gymnasium environments**
- [Replay buffers](https://github.com/thu-ml/tianshou/tree/master/tianshou/data/buffer): `tianshou/data/buffer`
- [Batch data](https://github.com/thu-ml/tianshou/blob/master/tianshou/data/batch.py): `tianshou/data/batch.py`

### Results

- [Experiment logging](https://github.com/thu-ml/tianshou/blob/master/tianshou/highlevel/logger.py): `tianshou/highlevel/logger.py`
- [RL evaluation](https://github.com/thu-ml/tianshou/blob/master/tianshou/evaluation/rliable_evaluation.py): `tianshou/evaluation/rliable_evaluation.py`
- [TensorBoard and W&B](https://github.com/thu-ml/tianshou/tree/master/tianshou/utils/logger): `tianshou/utils/logger`

### Other components

- **Researcher or app**

## Connections

- Researcher or app → Experiment setup: configures experiment
- Researcher or app → Algorithm interface: uses algorithm API
- Experiment setup → Training loop: runs training
- Experiment setup → Vector environments: creates environments
- Training loop → Experience collection: collects experience
- Training loop → Algorithm interface: requests updates
- Algorithm interface → Model-free RL: implements
- Algorithm interface → Imitation RL: implements
- Algorithm interface → Model-based RL: implements
- Algorithm interface → Multi-agent RL: implements
- Model-free RL → Actor and critic nets: uses networks
- Experience collection → Algorithm interface: gets actions
- Experience collection → Vector environments: steps environments
- Vector environments → Environment workers: dispatches work
- Environment workers → Gymnasium environments: interacts with
- Experience collection → Replay buffers: stores experience
- Algorithm interface → Replay buffers: samples experience
- Experience collection → Batch data: produces batches
- Training loop → Experiment logging: reports metrics
- Experiment logging → TensorBoard and W&B: writes logs
- RL evaluation → Experiment logging: reads experiment data