How volcengine/verl works
1. What this repository is
verl is a distributed post-training/RL library for LLMs and VLMs. Its architecture is built around a trainer control plane that coordinates multiple specialized worker types: model engines, rollout/inference engines, and reward computation. The repo is designed to swap backends without changing the overall RL dataflow.
2. Primary execution flow
The main runtime path is: dataset/input prompts → rollout generation → reward evaluation → advantage/return computation → policy/critic update → checkpoint/logging. That loop is implemented mainly under verl/trainer, with the component-specific work delegated to verl/workers/*.
3. Trainer layer
verl/trainer contains the top-level algorithms and entrypoints: PPO, GRPO-related flows, SFT, diffusion training, distillation, eval, and generation server modes. main_ppo.py, main_flowgrpo.py, main_generation_server.py, main_eval.py, and sft_trainer*.py are the main orchestration entrypoints. Configuration is heavily YAML-driven through verl/trainer/config, where algorithm, model, engine, rollout, reward, optimizer, and profiler settings are composed.
4. Worker architecture
The verl/workers tree is the main implementation boundary. It splits into:
- engine: training/inference-capable model wrappers
- rollout: generation services and adapters
- reward_manager: reward calculation and filtering
- config: typed config objects for each worker role
This separation is the core architectural boundary for diagramming.
5. Model engine backends
Under verl/workers/engine, the library supports multiple training backends: FSDP/FSDP2, Megatron, MindSpeed, TorchTitan, VeOmni, and Automodel. These implementations are abstracted behind common engine APIs, so the trainer can keep the same algorithm logic while changing sharding, parallelism, or optimization strategy.
6. Rollout / inference backends
Under verl/workers/rollout, generation can come from HF Transformers, vLLM, SGLang, or TensorRT-LLM. Some rollout paths are synchronous, while others are async server-based. This means rollout is not just a helper call; it is a first-class service boundary that can run on separate resources from training.
7. Reward subsystem
Reward logic is split between verl/workers/reward_manager and reusable scoring utilities in verl/utils/reward_score. The repo supports both function/verifiable rewards (e.g. math, code, exact-match style tasks) and model-based rewards. Reward managers also encapsulate policy-specific behaviors such as DAPO/PRIME-style processing or batch/naive variants.
8. Model and framework integration layer
verl/models and verl/utils/model* contain the compatibility layer that bridges Hugging Face model definitions into the supported training/inference backends. There are backend-specific model adapters for transformer families, Qwen variants, multimodal models, and mcore/Megatron integration. This layer is where model conversion, weight loading/saving, and backend-specific patches live.
9. Distributed orchestration
The repo uses a Ray-based distributed control model in many paths, with a lower-level verl/single_controller abstraction for worker groups and decorators. This gives the library a way to schedule colocated or separated actor/critic/rollout workers across GPUs, nodes, or specialized clusters.
10. Experimental advanced flows
verl/experimental contains next-generation architectures that extend the core loop:
- agent_loop: tool-calling and multi-turn agent training
- reward_loop: reward-model or reward-service loops
- fully_async_policy and one_step_off_policy: asynchronous/off-policy training variants
- separation: split execution and resource isolation
- teacher_loop: teacher/student style workflows
These are not just examples; they are alternate execution models layered on top of the same core abstractions.
11. Tooling and external environment integration
verl/tools provides tool schemas and adapters for search, sandbox execution, MCP clients, and task-specific tools such as GSM8K and Geo3K. This is the bridge from RL training into interactive environments, especially for agentic/multi-turn and tool-use recipes.
12. State, checkpointing, and persistence
Checkpoint handling is split across verl/utils/checkpoint and verl/checkpoint_engine. The code supports backend-aware checkpoint engines for distributed training stacks, so persistence is coupled to the training backend and parallelism strategy rather than being a generic save/load layer.
13. Data and preprocessing boundaries
verl/utils/dataset holds shared dataset primitives, collation, and RL/SFT dataset logic. The examples/data_preprocess scripts show that the repo expects users to prepare task-specific training data outside the core training loop, then feed it into the trainer through structured datasets and configs.
14. Ops, packaging, and validation
The examples/ tree provides runnable recipes for many model/algorithm/backend combinations, while docker/ and GitHub Actions encode a large hardware matrix: NVIDIA, AMD/ROCm, and Ascend/NPU. The tests/ tree mirrors the architecture with dedicated coverage for trainers, workers, distributed control, rollouts, reward loops, and hardware-specific behaviors.
15. How to draw the architecture
A good diagram should show a central trainer/orchestrator box connected to three major service planes: training engines, rollout engines, and reward managers. Around that, show supporting layers for model/backends, datasets/preprocessing, checkpointing, tools/agent environments, and distributed runtime (Ray/single-controller), with experimental async/off-policy flows as optional branches off the main loop.
Main entry points: engine Engines · trainer Trainer · ppo PPO core · rollout Rollouts
Components 27
Trainer plane
Top-level orchestration, algorithm entrypoints, and config composition.
- Trainer · Main RL/SFT control plane coordinating dataflow and updates.
verl/trainer - Config · Algorithm, model, rollout, reward, optimizer, and profiler composition.
verl/trainer/config - PPO core · Core PPO/GRPO-style training logic and metric/reward handling.
verl/trainer/ppo - SFT · Supervised fine-tuning orchestration paths.
verl/trainer/sft_trainer.py - Diffusion · Diffusion training and metric utilities.
verl/trainer/diffusion - Distillation · On-policy and backend-specific distillation losses and flows.
verl/trainer/distillation - Eval/Serve · Evaluation and generation-server entrypoints.
verl/trainer/main_eval.py
Worker plane
Execution backends for engines, rollouts, and rewards.
- Engines · Training/inference-capable model engine boundary.
verl/workers/engine - FSDP · FSDP/FSDP2-style sharded training backend.
verl/workers/engine/fsdp - Megatron · Megatron-style tensor/pipeline parallel backend.
verl/workers/engine/megatron - MindSpeed · Ascend-oriented training backend.
verl/workers/engine/mindspeed - TorchTitan · TorchTitan backend adapter for training engines.
verl/workers/engine/torchtitan - VeOmni · VeOmni backend adapter for distributed model execution.
verl/workers/engine/veomni - Rollouts · First-class inference/rollout boundary used during training.
verl/workers/rollout - vLLM · vLLM-based generation and async serving paths.
verl/workers/rollout/vllm_rollout - SGLang · SGLang-based generation and HTTP server rollout paths.
verl/workers/rollout/sglang_rollout - TensorRT-LLM · TensorRT-LLM rollout and async server integration.
verl/workers/rollout/trtllm_rollout - Rewards · Reward calculation, filtering, and policy-specific processing.
verl/workers/reward_manager
Support layers
Shared model, data, distributed runtime, checkpointing, and tools.
- Model adapters · Hugging Face to backend model bridging, patches, and loaders.
verl/models - Datasets · Shared RL/SFT dataset, collation, and preprocessing primitives.
verl/utils/dataset - Checkpointing · Backend-aware checkpoint managers and save/load orchestration.
verl/utils/checkpoint - Ray/control · Ray-based distributed control and single-controller worker groups.
verl/single_controller - Tools · Tool schemas, search, sandbox, MCP, and task-specific tool adapters.
verl/tools
Experimental flows
Alternate execution models layered on top of the core abstractions.
- Experimental · Alternate execution models extending the main training loop.
verl/experimental - Async/off-policy · Fully async and one-step off-policy training variants.
verl/experimental/fully_async_policy - Reward loops · Reward-model and reward-service loops with alternate routers.
verl/experimental/reward_loop - Agent loops · Tool-calling and multi-turn agent training loops.
verl/experimental/agent_loop
Connections 33
- Trainer → Config: loads · Trainer entrypoints compose algorithm and backend settings from YAML.
- Trainer → PPO core: runs · Main orchestration delegates core RL optimization to PPO/GRPO logic.
- Trainer → SFT: runs · Trainer also drives supervised fine-tuning flows.
- Trainer → Diffusion: runs · Trainer can execute diffusion training variants.
- Trainer → Distillation: runs · Trainer supports distillation workflows.
- Trainer → Eval/Serve: serves · Entrypoints include evaluation and generation-server modes.
- PPO core → Engines: updates · Policy/critic updates are executed through model engines.
- PPO core → Rollouts: samples · Training loop requests prompt rollout generation.
- PPO core → Rewards: scores · Rewards are computed and filtered before advantages/returns.
- PPO core → Datasets: consumes · Prompts and trajectories come from dataset utilities.
- PPO core → Checkpointing: persists · Checkpoints and training state are saved around the loop.
- Engines → FSDP: backend · FSDP is one concrete engine implementation.
- Engines → Megatron: backend · Megatron is one concrete engine implementation.
- Engines → MindSpeed: backend · MindSpeed is one concrete engine implementation.
- Engines → TorchTitan: backend · TorchTitan is one concrete engine implementation.
- Engines → VeOmni: backend · VeOmni is one concrete engine implementation.
- Engines → Model adapters: binds · Engines consume model adapters and backend-specific patches.
- Rollouts → vLLM: backend · vLLM provides one rollout/inference backend.
- Rollouts → SGLang: backend · SGLang provides one rollout/inference backend.
- Rollouts → TensorRT-LLM: backend · TensorRT-LLM provides one rollout/inference backend.
- Rollouts → Model adapters: loads · Rollout backends rely on shared model conversion/loading paths.
- Rewards → Tools: uses · Rewarding can depend on verifiable task tools and sandbox helpers.
- Rewards → Datasets: filters · Reward managers operate on sampled batches and completions.
- Model adapters → Checkpointing: saves/loads · Model conversion and backend adapters interact with persistence.
- Ray/control → Trainer: hosts · Distributed controller schedules trainer-side workers and roles.
- Ray/control → Engines: schedules · Worker groups place engines across GPUs/nodes.
- Ray/control → Rollouts: schedules · Rollout services can run colocated or separately.
- Experimental → Async/off-policy: includes · Experimental namespace contains async/off-policy variants.
- Experimental → Reward loops: includes · Experimental namespace contains reward-loop variants.
- Experimental → Agent loops: includes · Experimental namespace contains agent-loop variants.
- Async/off-policy → Trainer: extends · Async/off-policy flows reuse the main trainer abstractions.
- Reward loops → Rewards: extends · Experimental reward loops reuse and specialize reward processing.
- Agent loops → Tools: extends · Agent loops depend on tool schemas and environment bridges.