# verl-project/verl architecture

> Architecture diagram of the GitHub repository verl-project/verl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/verl-project/verl
- Repository: https://github.com/verl-project/verl

## Overview

verl is a distributed reinforcement-learning and post-training library for LLMs. The principal workflow configures a training run, prepares prompt data, generates rollouts, scores responses, and updates policy models; checkpoints preserve trained state. Its documented scope also includes SFT, evaluation, distillation, tool-using agents, asynchronous training, and modular compute backends. The README documents hybrid-controller dataflows and backend integrations, but sampled source excerpts are incomplete, so links between unsampled subsystems are limited to those documented capabilities.

Main entry points: [`verl/trainer/ppo/ray_trainer.py`](https://github.com/verl-project/verl/blob/main/verl/trainer/ppo/ray_trainer.py) (PPO trainer), [`verl/workers/engine_workers.py`](https://github.com/verl-project/verl/blob/main/verl/workers/engine_workers.py) (Engine workers), [`verl/workers/rollout`](https://github.com/verl-project/verl/tree/main/verl/workers/rollout) (Rollout workers), [`verl/single_controller`](https://github.com/verl-project/verl/tree/main/verl/single_controller) (Worker controller)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_training["Training workflows"]
  node_ppo_entry["PPO entry<br/>[main_ppo.py]"]
  node_ppo_trainer["PPO trainer<br/>[ray_trainer.py]"]
  node_datasets["RL datasets<br/>[rl_dataset.py]"]
  node_reward["Reward scoring"]
  node_algorithms["RL algorithms<br/>[core_algos.py]"]
  node_sft["SFT training<br/>[sft_trainer.py]"]
  node_evaluation["Evaluation<br/>[main_eval.py]"]
end

subgraph group_execution["Distributed execution"]
  node_rollout["Rollout workers"]
  node_worker_controller["Worker controller"]
  node_engine_workers["Engine workers<br/>[engine_workers.py]"]
end

subgraph group_models["Model and state"]
  node_engines["Training engines"]
  node_models["Model adapters"]
  node_checkpoint["Checkpoint managers"]
  node_checkpoint_transfer["Checkpoint transfer"]
  node_model_store[("Checkpoint storage")]
end

subgraph group_extensions["Post-training extensions"]
  node_distillation["Distillation"]
  node_agent_loops["Agent loops"]
  node_async_policy["Async policy"]
  node_reward_loop["Reward loop"]
  node_tool_api["Tool APIs"]
end

node_engineer(("Training user"))
node_llm_backends["LLM frameworks"]
node_cluster["Ray cluster"]

node_engineer -->|"configures run"| node_ppo_entry
node_ppo_entry -->|"starts training"| node_ppo_trainer
node_ppo_trainer -->|"loads prompts"| node_datasets
node_ppo_trainer -->|"dispatches work"| node_worker_controller
node_worker_controller -->|"invokes workers"| node_engine_workers
node_engine_workers -->|"requests generation"| node_rollout
node_rollout -->|"scores responses"| node_reward
node_ppo_trainer -->|"computes updates"| node_algorithms
node_engine_workers -->|"runs computation"| node_engines
node_engines -->|"executes models"| node_models
node_ppo_trainer -->|"saves state"| node_checkpoint
node_checkpoint -->|"writes checkpoints"| node_model_store
node_engine_workers -->|"syncs weights"| node_checkpoint_transfer
node_checkpoint_transfer -->|"updates weights"| node_rollout
node_engines -.->|"integrates backends"| node_llm_backends
node_worker_controller -.->|"distributes workers"| node_cluster
node_engineer -->|"starts SFT"| node_sft
node_engineer -->|"starts evaluation"| node_evaluation
node_engineer -->|"configures distillation"| node_distillation
node_ppo_trainer -.->|"supports async training"| node_async_policy
node_rollout -.->|"supports agent flows"| node_agent_loops
node_agent_loops -->|"invokes tools"| node_tool_api
node_ppo_trainer -.->|"supports reward workflows"| node_reward_loop

click node_ppo_entry "https://github.com/verl-project/verl/blob/main/verl/trainer/main_ppo.py"
click node_ppo_trainer "https://github.com/verl-project/verl/blob/main/verl/trainer/ppo/ray_trainer.py"
click node_datasets "https://github.com/verl-project/verl/blob/main/verl/utils/dataset/rl_dataset.py"
click node_rollout "https://github.com/verl-project/verl/tree/main/verl/workers/rollout"
click node_reward "https://github.com/verl-project/verl/tree/main/verl/workers/reward_manager"
click node_algorithms "https://github.com/verl-project/verl/blob/main/verl/trainer/ppo/core_algos.py"
click node_worker_controller "https://github.com/verl-project/verl/tree/main/verl/single_controller"
click node_engine_workers "https://github.com/verl-project/verl/blob/main/verl/workers/engine_workers.py"
click node_engines "https://github.com/verl-project/verl/tree/main/verl/workers/engine"
click node_models "https://github.com/verl-project/verl/tree/main/verl/models"
click node_checkpoint "https://github.com/verl-project/verl/tree/main/verl/utils/checkpoint"
click node_checkpoint_transfer "https://github.com/verl-project/verl/tree/main/verl/checkpoint_engine"
click node_sft "https://github.com/verl-project/verl/blob/main/verl/trainer/sft_trainer.py"
click node_evaluation "https://github.com/verl-project/verl/blob/main/verl/trainer/main_eval.py"
click node_distillation "https://github.com/verl-project/verl/tree/main/verl/trainer/distillation"
click node_agent_loops "https://github.com/verl-project/verl/tree/main/verl/experimental/agent_loop"
click node_async_policy "https://github.com/verl-project/verl/tree/main/verl/experimental/fully_async_policy"
click node_reward_loop "https://github.com/verl-project/verl/tree/main/verl/experimental/reward_loop"
click node_tool_api "https://github.com/verl-project/verl/tree/main/verl/tools"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ppo_entry,node_ppo_trainer,node_datasets,node_reward,node_algorithms,node_sft,node_evaluation,node_engineer toneBlue
class node_rollout,node_worker_controller,node_engine_workers toneAmber
class node_engines,node_models,node_checkpoint,node_checkpoint_transfer,node_model_store toneMint
class node_distillation,node_agent_loops,node_async_policy,node_reward_loop,node_tool_api toneRose
class node_llm_backends,node_cluster toneIndigo
```

## Components

### Training workflows

- [PPO entry](https://github.com/verl-project/verl/blob/main/verl/trainer/main_ppo.py): `verl/trainer/main_ppo.py`
- [PPO trainer](https://github.com/verl-project/verl/blob/main/verl/trainer/ppo/ray_trainer.py): `verl/trainer/ppo/ray_trainer.py`
- [RL datasets](https://github.com/verl-project/verl/blob/main/verl/utils/dataset/rl_dataset.py): `verl/utils/dataset/rl_dataset.py`
- [Reward scoring](https://github.com/verl-project/verl/tree/main/verl/workers/reward_manager): `verl/workers/reward_manager`
- [RL algorithms](https://github.com/verl-project/verl/blob/main/verl/trainer/ppo/core_algos.py): `verl/trainer/ppo/core_algos.py`
- [SFT training](https://github.com/verl-project/verl/blob/main/verl/trainer/sft_trainer.py): `verl/trainer/sft_trainer.py`
- [Evaluation](https://github.com/verl-project/verl/blob/main/verl/trainer/main_eval.py): `verl/trainer/main_eval.py`

### Distributed execution

- [Rollout workers](https://github.com/verl-project/verl/tree/main/verl/workers/rollout): `verl/workers/rollout`
- [Worker controller](https://github.com/verl-project/verl/tree/main/verl/single_controller): `verl/single_controller`
- [Engine workers](https://github.com/verl-project/verl/blob/main/verl/workers/engine_workers.py): `verl/workers/engine_workers.py`

### Model and state

- [Training engines](https://github.com/verl-project/verl/tree/main/verl/workers/engine): `verl/workers/engine`
- [Model adapters](https://github.com/verl-project/verl/tree/main/verl/models): `verl/models`
- [Checkpoint managers](https://github.com/verl-project/verl/tree/main/verl/utils/checkpoint): `verl/utils/checkpoint`
- [Checkpoint transfer](https://github.com/verl-project/verl/tree/main/verl/checkpoint_engine): `verl/checkpoint_engine`
- **Checkpoint storage**

### Post-training extensions

- [Distillation](https://github.com/verl-project/verl/tree/main/verl/trainer/distillation): `verl/trainer/distillation`
- [Agent loops](https://github.com/verl-project/verl/tree/main/verl/experimental/agent_loop): `verl/experimental/agent_loop`
- [Async policy](https://github.com/verl-project/verl/tree/main/verl/experimental/fully_async_policy): `verl/experimental/fully_async_policy`
- [Reward loop](https://github.com/verl-project/verl/tree/main/verl/experimental/reward_loop): `verl/experimental/reward_loop`
- [Tool APIs](https://github.com/verl-project/verl/tree/main/verl/tools): `verl/tools`

### Other components

- **Training user**
- **LLM frameworks**
- **Ray cluster**

## Connections

- Training user → PPO entry: configures run
- PPO entry → PPO trainer: starts training
- PPO trainer → RL datasets: loads prompts
- PPO trainer → Worker controller: dispatches work
- Worker controller → Engine workers: invokes workers
- Engine workers → Rollout workers: requests generation
- Rollout workers → Reward scoring: scores responses
- PPO trainer → RL algorithms: computes updates
- Engine workers → Training engines: runs computation
- Training engines → Model adapters: executes models
- PPO trainer → Checkpoint managers: saves state
- Checkpoint managers → Checkpoint storage: writes checkpoints
- Engine workers → Checkpoint transfer: syncs weights
- Checkpoint transfer → Rollout workers: updates weights
- Training engines → LLM frameworks: integrates backends
- Worker controller → Ray cluster: distributes workers
- Training user → SFT training: starts SFT
- Training user → Evaluation: starts evaluation
- Training user → Distillation: configures distillation
- PPO trainer → Async policy: supports async training
- Rollout workers → Agent loops: supports agent flows
- Agent loops → Tool APIs: invokes tools
- PPO trainer → Reward loop: supports reward workflows
