# volcengine/verl architecture

> Architecture diagram of the GitHub repository volcengine/verl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/volcengine/verl
- Repository: https://github.com/volcengine/verl

## Overview

1. **What this repository is**
   verl is a distributed post-training/RL library for LLMs and VLMs. Its architecture is built around a **trainer control plane** that coordinates multiple specialized worker types: model engines, rollout/inference engines, and reward computation. The repo is designed to swap backends without changing the overall RL dataflow.

2. **Primary execution flow**
   The main runtime path is: **dataset/input prompts → rollout generation → reward evaluation → advantage/return computation → policy/critic update → checkpoint/logging**. That loop is implemented mainly under `verl/trainer`, with the component-specific work delegated to `verl/workers/*`.

3. **Trainer layer**
   `verl/trainer` contains the top-level algorithms and entrypoints: PPO, GRPO-related flows, SFT, diffusion training, distillation, eval, and generation server modes. `main_ppo.py`, `main_flowgrpo.py`, `main_generation_server.py`, `main_eval.py`, and `sft_trainer*.py` are the main orchestration entrypoints. Configuration is heavily YAML-driven through `verl/trainer/config`, where algorithm, model, engine, rollout, reward, optimizer, and profiler settings are composed.

4. **Worker architecture**
   The `verl/workers` tree is the main implementation boundary. It splits into:
   - `engine`: training/inference-capable model wrappers
   - `rollout`: generation services and adapters
   - `reward_manager`: reward calculation and filtering
   - `config`: typed config objects for each worker role  
   This separation is the core architectural boundary for diagramming.

5. **Model engine backends**
   Under `verl/workers/engine`, the library supports multiple training backends: **FSDP/FSDP2, Megatron, MindSpeed, TorchTitan, VeOmni, and Automodel**. These implementations are abstracted behind common engine APIs, so the trainer can keep the same algorithm logic while changing sharding, parallelism, or optimization strategy.

6. **Rollout / inference backends**
   Under `verl/workers/rollout`, generation can come from **HF Transformers**, **vLLM**, **SGLang**, or **TensorRT-LLM**. Some rollout paths are synchronous, while others are async server-based. This means rollout is not just a helper call; it is a first-class service boundary that can run on separate resources from training.

7. **Reward subsystem**
   Reward logic is split between `verl/workers/reward_manager` and reusable scoring utilities in `verl/utils/reward_score`. The repo supports both **function/verifiable rewards** (e.g. math, code, exact-match style tasks) and **model-based rewards**. Reward managers also encapsulate policy-specific behaviors such as DAPO/PRIME-style processing or batch/naive variants.

8. **Model and framework integration layer**
   `verl/models` and `verl/utils/model*` contain the compatibility layer that bridges Hugging Face model definitions into the supported training/inference backends. There are backend-specific model adapters for transformer families, Qwen variants, multimodal models, and mcore/Megatron integration. This layer is where model conversion, weight loading/saving, and backend-specific patches live.

9. **Distributed orchestration**
   The repo uses a **Ray-based distributed control model** in many paths, with a lower-level `verl/single_controller` abstraction for worker groups and decorators. This gives the library a way to schedule colocated or separated actor/critic/rollout workers across GPUs, nodes, or specialized clusters.

10. **Experimental advanced flows**
   `verl/experimental` contains next-generation architectures that extend the core loop:
   - `agent_loop`: tool-calling and multi-turn agent training
   - `reward_loop`: reward-model or reward-service loops
   - `fully_async_policy` and `one_step_off_policy`: asynchronous/off-policy training variants
   - `separation`: split execution and resource isolation
   - `teacher_loop`: teacher/student style workflows  
   These are not just examples; they are alternate execution models layered on top of the same core abstractions.

11. **Tooling and external environment integration**
   `verl/tools` provides tool schemas and adapters for search, sandbox execution, MCP clients, and task-specific tools such as GSM8K and Geo3K. This is the bridge from RL training into interactive environments, especially for agentic/multi-turn and tool-use recipes.

12. **State, checkpointing, and persistence**
   Checkpoint handling is split across `verl/utils/checkpoint` and `verl/checkpoint_engine`. The code supports backend-aware checkpoint engines for distributed training stacks, so persistence is coupled to the training backend and parallelism strategy rather than being a generic save/load layer.

13. **Data and preprocessing boundaries**
   `verl/utils/dataset` holds shared dataset primitives, collation, and RL/SFT dataset logic. The `examples/data_preprocess` scripts show that the repo expects users to prepare task-specific training data outside the core training loop, then feed it into the trainer through structured datasets and configs.

14. **Ops, packaging, and validation**
   The `examples/` tree provides runnable recipes for many model/algorithm/backend combinations, while `docker/` and GitHub Actions encode a large hardware matrix: NVIDIA, AMD/ROCm, and Ascend/NPU. The `tests/` tree mirrors the architecture with dedicated coverage for trainers, workers, distributed control, rollouts, reward loops, and hardware-specific behaviors.

15. **How to draw the architecture**
   A good diagram should show a central **trainer/orchestrator** box connected to three major service planes: **training engines**, **rollout engines**, and **reward managers**. Around that, show supporting layers for **model/backends**, **datasets/preprocessing**, **checkpointing**, **tools/agent environments**, and **distributed runtime (Ray/single-controller)**, with experimental async/off-policy flows as optional branches off the main loop.

Main entry points: [`verl/workers/engine`](https://github.com/volcengine/verl/tree/main/verl/workers/engine) (Engines), [`verl/trainer`](https://github.com/volcengine/verl/tree/main/verl/trainer) (Trainer), [`verl/trainer/ppo`](https://github.com/volcengine/verl/tree/main/verl/trainer/ppo) (PPO core), [`verl/workers/rollout`](https://github.com/volcengine/verl/tree/main/verl/workers/rollout) (Rollouts)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_trainer["Trainer plane"]
  node_trainer_main["Trainer<br/>orchestrator"]
  node_trainer_config["Config"]
  node_trainer_ppo["PPO core<br/>rl algorithm"]
  node_trainer_sft["SFT<br/>supervised trainer<br/>[sft_trainer.py]"]
  node_trainer_diffusion["Diffusion<br/>trainer variant"]
  node_trainer_distill["Distillation<br/>trainer variant"]
  node_trainer_eval["Eval/Serve<br/>entrypoint<br/>[main_eval.py]"]
end

subgraph group_group_workers["Worker plane"]
  node_engine_base["Engines<br/>model runtime"]
  node_engine_fsdp["FSDP<br/>backend"]
  node_engine_megatron["Megatron<br/>backend"]
  node_engine_mindspeed["MindSpeed<br/>backend"]
  node_engine_torchtitan["TorchTitan<br/>backend"]
  node_engine_veomni["VeOmni<br/>backend"]
  node_rollout_base["Rollouts<br/>generation service"]
  node_rollout_vllm["vLLM<br/>rollout backend"]
  node_rollout_sglang["SGLang<br/>rollout backend"]
  node_rollout_trtllm["TensorRT-LLM<br/>rollout backend"]
  node_reward_mgr["Rewards<br/>reward manager"]
end

subgraph group_group_support["Support layers"]
  node_models_layer["Model adapters<br/>compatibility layer"]
  node_dataset_layer["Datasets<br/>data layer"]
  node_checkpoint_layer["Checkpointing<br/>persistence"]
  node_dist_runtime["Ray/control<br/>distributed runtime"]
  node_tooling_layer["Tools<br/>environment bridge"]
end

subgraph group_group_experimental["Experimental flows"]
  node_exp_flows["Experimental<br/>advanced flows"]
  node_exp_async["Async/off-policy<br/>experimental flow"]
  node_exp_reward["Reward loops<br/>experimental flow"]
  node_exp_agent["Agent loops<br/>experimental flow"]
end

node_trainer_main -->|"loads"| node_trainer_config
node_trainer_main -->|"runs"| node_trainer_ppo
node_trainer_main -->|"runs"| node_trainer_sft
node_trainer_main -->|"runs"| node_trainer_diffusion
node_trainer_main -->|"runs"| node_trainer_distill
node_trainer_main -->|"serves"| node_trainer_eval
node_trainer_ppo -->|"updates"| node_engine_base
node_trainer_ppo -->|"samples"| node_rollout_base
node_trainer_ppo -->|"scores"| node_reward_mgr
node_trainer_ppo -->|"consumes"| node_dataset_layer
node_trainer_ppo -->|"persists"| node_checkpoint_layer
node_engine_base -->|"backend"| node_engine_fsdp
node_engine_base -->|"backend"| node_engine_megatron
node_engine_base -->|"backend"| node_engine_mindspeed
node_engine_base -->|"backend"| node_engine_torchtitan
node_engine_base -->|"backend"| node_engine_veomni
node_engine_base -->|"binds"| node_models_layer
node_rollout_base -->|"backend"| node_rollout_vllm
node_rollout_base -->|"backend"| node_rollout_sglang
node_rollout_base -->|"backend"| node_rollout_trtllm
node_rollout_base -->|"loads"| node_models_layer
node_reward_mgr -->|"uses"| node_tooling_layer
node_reward_mgr -->|"filters"| node_dataset_layer
node_models_layer -->|"saves/loads"| node_checkpoint_layer
node_dist_runtime -->|"hosts"| node_trainer_main
node_dist_runtime -->|"schedules"| node_engine_base
node_dist_runtime -->|"schedules"| node_rollout_base
node_exp_flows -->|"includes"| node_exp_async
node_exp_flows -->|"includes"| node_exp_reward
node_exp_flows -->|"includes"| node_exp_agent
node_exp_async -.->|"extends"| node_trainer_main
node_exp_reward -.->|"extends"| node_reward_mgr
node_exp_agent -.->|"extends"| node_tooling_layer

click node_trainer_main "https://github.com/volcengine/verl/tree/main/verl/trainer"
click node_trainer_config "https://github.com/volcengine/verl/tree/main/verl/trainer/config"
click node_trainer_ppo "https://github.com/volcengine/verl/tree/main/verl/trainer/ppo"
click node_trainer_sft "https://github.com/volcengine/verl/blob/main/verl/trainer/sft_trainer.py"
click node_trainer_diffusion "https://github.com/volcengine/verl/tree/main/verl/trainer/diffusion"
click node_trainer_distill "https://github.com/volcengine/verl/tree/main/verl/trainer/distillation"
click node_trainer_eval "https://github.com/volcengine/verl/blob/main/verl/trainer/main_eval.py"
click node_engine_base "https://github.com/volcengine/verl/tree/main/verl/workers/engine"
click node_engine_fsdp "https://github.com/volcengine/verl/tree/main/verl/workers/engine/fsdp"
click node_engine_megatron "https://github.com/volcengine/verl/tree/main/verl/workers/engine/megatron"
click node_engine_mindspeed "https://github.com/volcengine/verl/tree/main/verl/workers/engine/mindspeed"
click node_engine_torchtitan "https://github.com/volcengine/verl/tree/main/verl/workers/engine/torchtitan"
click node_engine_veomni "https://github.com/volcengine/verl/tree/main/verl/workers/engine/veomni"
click node_rollout_base "https://github.com/volcengine/verl/tree/main/verl/workers/rollout"
click node_rollout_vllm "https://github.com/volcengine/verl/tree/main/verl/workers/rollout/vllm_rollout"
click node_rollout_sglang "https://github.com/volcengine/verl/tree/main/verl/workers/rollout/sglang_rollout"
click node_rollout_trtllm "https://github.com/volcengine/verl/tree/main/verl/workers/rollout/trtllm_rollout"
click node_reward_mgr "https://github.com/volcengine/verl/tree/main/verl/workers/reward_manager"
click node_models_layer "https://github.com/volcengine/verl/tree/main/verl/models"
click node_dataset_layer "https://github.com/volcengine/verl/tree/main/verl/utils/dataset"
click node_checkpoint_layer "https://github.com/volcengine/verl/tree/main/verl/utils/checkpoint"
click node_dist_runtime "https://github.com/volcengine/verl/tree/main/verl/single_controller"
click node_tooling_layer "https://github.com/volcengine/verl/tree/main/verl/tools"
click node_exp_flows "https://github.com/volcengine/verl/tree/main/verl/experimental"
click node_exp_async "https://github.com/volcengine/verl/tree/main/verl/experimental/fully_async_policy"
click node_exp_reward "https://github.com/volcengine/verl/tree/main/verl/experimental/reward_loop"
click node_exp_agent "https://github.com/volcengine/verl/tree/main/verl/experimental/agent_loop"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_trainer_main,node_trainer_config,node_trainer_ppo,node_trainer_sft,node_trainer_diffusion,node_trainer_distill,node_trainer_eval toneBlue
class node_engine_base,node_engine_fsdp,node_engine_megatron,node_engine_mindspeed,node_engine_torchtitan,node_engine_veomni,node_rollout_base,node_rollout_vllm,node_rollout_sglang,node_rollout_trtllm,node_reward_mgr toneAmber
class node_models_layer,node_dataset_layer,node_checkpoint_layer,node_dist_runtime,node_tooling_layer toneMint
class node_exp_flows,node_exp_async,node_exp_reward,node_exp_agent toneRose
```

## Components

### Trainer plane

Top-level orchestration, algorithm entrypoints, and config composition.

- [Trainer](https://github.com/volcengine/verl/tree/main/verl/trainer): orchestrator · Main RL/SFT control plane coordinating dataflow and updates. · `verl/trainer`
- [Config](https://github.com/volcengine/verl/tree/main/verl/trainer/config): yaml config · Algorithm, model, rollout, reward, optimizer, and profiler composition. · `verl/trainer/config`
- [PPO core](https://github.com/volcengine/verl/tree/main/verl/trainer/ppo): rl algorithm · Core PPO/GRPO-style training logic and metric/reward handling. · `verl/trainer/ppo`
- [SFT](https://github.com/volcengine/verl/blob/main/verl/trainer/sft_trainer.py): supervised trainer · Supervised fine-tuning orchestration paths. · `verl/trainer/sft_trainer.py`
- [Diffusion](https://github.com/volcengine/verl/tree/main/verl/trainer/diffusion): trainer variant · Diffusion training and metric utilities. · `verl/trainer/diffusion`
- [Distillation](https://github.com/volcengine/verl/tree/main/verl/trainer/distillation): trainer variant · On-policy and backend-specific distillation losses and flows. · `verl/trainer/distillation`
- [Eval/Serve](https://github.com/volcengine/verl/blob/main/verl/trainer/main_eval.py): entrypoint · Evaluation and generation-server entrypoints. · `verl/trainer/main_eval.py`

### Worker plane

Execution backends for engines, rollouts, and rewards.

- [Engines](https://github.com/volcengine/verl/tree/main/verl/workers/engine): model runtime · Training/inference-capable model engine boundary. · `verl/workers/engine`
- [FSDP](https://github.com/volcengine/verl/tree/main/verl/workers/engine/fsdp): backend · FSDP/FSDP2-style sharded training backend. · `verl/workers/engine/fsdp`
- [Megatron](https://github.com/volcengine/verl/tree/main/verl/workers/engine/megatron): backend · Megatron-style tensor/pipeline parallel backend. · `verl/workers/engine/megatron`
- [MindSpeed](https://github.com/volcengine/verl/tree/main/verl/workers/engine/mindspeed): backend · Ascend-oriented training backend. · `verl/workers/engine/mindspeed`
- [TorchTitan](https://github.com/volcengine/verl/tree/main/verl/workers/engine/torchtitan): backend · TorchTitan backend adapter for training engines. · `verl/workers/engine/torchtitan`
- [VeOmni](https://github.com/volcengine/verl/tree/main/verl/workers/engine/veomni): backend · VeOmni backend adapter for distributed model execution. · `verl/workers/engine/veomni`
- [Rollouts](https://github.com/volcengine/verl/tree/main/verl/workers/rollout): generation service · First-class inference/rollout boundary used during training. · `verl/workers/rollout`
- [vLLM](https://github.com/volcengine/verl/tree/main/verl/workers/rollout/vllm_rollout): rollout backend · vLLM-based generation and async serving paths. · `verl/workers/rollout/vllm_rollout`
- [SGLang](https://github.com/volcengine/verl/tree/main/verl/workers/rollout/sglang_rollout): rollout backend · SGLang-based generation and HTTP server rollout paths. · `verl/workers/rollout/sglang_rollout`
- [TensorRT-LLM](https://github.com/volcengine/verl/tree/main/verl/workers/rollout/trtllm_rollout): rollout backend · TensorRT-LLM rollout and async server integration. · `verl/workers/rollout/trtllm_rollout`
- [Rewards](https://github.com/volcengine/verl/tree/main/verl/workers/reward_manager): reward manager · Reward calculation, filtering, and policy-specific processing. · `verl/workers/reward_manager`

### Support layers

Shared model, data, distributed runtime, checkpointing, and tools.

- [Model adapters](https://github.com/volcengine/verl/tree/main/verl/models): compatibility layer · Hugging Face to backend model bridging, patches, and loaders. · `verl/models`
- [Datasets](https://github.com/volcengine/verl/tree/main/verl/utils/dataset): data layer · Shared RL/SFT dataset, collation, and preprocessing primitives. · `verl/utils/dataset`
- [Checkpointing](https://github.com/volcengine/verl/tree/main/verl/utils/checkpoint): persistence · Backend-aware checkpoint managers and save/load orchestration. · `verl/utils/checkpoint`
- [Ray/control](https://github.com/volcengine/verl/tree/main/verl/single_controller): distributed runtime · Ray-based distributed control and single-controller worker groups. · `verl/single_controller`
- [Tools](https://github.com/volcengine/verl/tree/main/verl/tools): environment bridge · Tool schemas, search, sandbox, MCP, and task-specific tool adapters. · `verl/tools`

### Experimental flows

Alternate execution models layered on top of the core abstractions.

- [Experimental](https://github.com/volcengine/verl/tree/main/verl/experimental): advanced flows · Alternate execution models extending the main training loop. · `verl/experimental`
- [Async/off-policy](https://github.com/volcengine/verl/tree/main/verl/experimental/fully_async_policy): experimental flow · Fully async and one-step off-policy training variants. · `verl/experimental/fully_async_policy`
- [Reward loops](https://github.com/volcengine/verl/tree/main/verl/experimental/reward_loop): experimental flow · Reward-model and reward-service loops with alternate routers. · `verl/experimental/reward_loop`
- [Agent loops](https://github.com/volcengine/verl/tree/main/verl/experimental/agent_loop): experimental flow · Tool-calling and multi-turn agent training loops. · `verl/experimental/agent_loop`

## Connections

- Trainer → Config: loads · Trainer entrypoints compose algorithm and backend settings from YAML.
- Trainer → PPO core: runs · Main orchestration delegates core RL optimization to PPO/GRPO logic.
- Trainer → SFT: runs · Trainer also drives supervised fine-tuning flows.
- Trainer → Diffusion: runs · Trainer can execute diffusion training variants.
- Trainer → Distillation: runs · Trainer supports distillation workflows.
- Trainer → Eval/Serve: serves · Entrypoints include evaluation and generation-server modes.
- PPO core → Engines: updates · Policy/critic updates are executed through model engines.
- PPO core → Rollouts: samples · Training loop requests prompt rollout generation.
- PPO core → Rewards: scores · Rewards are computed and filtered before advantages/returns.
- PPO core → Datasets: consumes · Prompts and trajectories come from dataset utilities.
- PPO core → Checkpointing: persists · Checkpoints and training state are saved around the loop.
- Engines → FSDP: backend · FSDP is one concrete engine implementation.
- Engines → Megatron: backend · Megatron is one concrete engine implementation.
- Engines → MindSpeed: backend · MindSpeed is one concrete engine implementation.
- Engines → TorchTitan: backend · TorchTitan is one concrete engine implementation.
- Engines → VeOmni: backend · VeOmni is one concrete engine implementation.
- Engines → Model adapters: binds · Engines consume model adapters and backend-specific patches.
- Rollouts → vLLM: backend · vLLM provides one rollout/inference backend.
- Rollouts → SGLang: backend · SGLang provides one rollout/inference backend.
- Rollouts → TensorRT-LLM: backend · TensorRT-LLM provides one rollout/inference backend.
- Rollouts → Model adapters: loads · Rollout backends rely on shared model conversion/loading paths.
- Rewards → Tools: uses · Rewarding can depend on verifiable task tools and sandbox helpers.
- Rewards → Datasets: filters · Reward managers operate on sampled batches and completions.
- Model adapters → Checkpointing: saves/loads · Model conversion and backend adapters interact with persistence.
- Ray/control → Trainer: hosts · Distributed controller schedules trainer-side workers and roles.
- Ray/control → Engines: schedules · Worker groups place engines across GPUs/nodes.
- Ray/control → Rollouts: schedules · Rollout services can run colocated or separately.
- Experimental → Async/off-policy: includes · Experimental namespace contains async/off-policy variants.
- Experimental → Reward loops: includes · Experimental namespace contains reward-loop variants.
- Experimental → Agent loops: includes · Experimental namespace contains agent-loop variants.
- Async/off-policy → Trainer: extends · Async/off-policy flows reuse the main trainer abstractions.
- Reward loops → Rewards: extends · Experimental reward loops reuse and specialize reward processing.
- Agent loops → Tools: extends · Agent loops depend on tool schemas and environment bridges.
