How verl-project/verl works
verl is a distributed reinforcement-learning and post-training library for LLMs. The principal workflow configures a training run, prepares prompt data, generates rollouts, scores responses, and updates policy models; checkpoints preserve trained state. Its documented scope also includes SFT, evaluation, distillation, tool-using agents, asynchronous training, and modular compute backends. The README documents hybrid-controller dataflows and backend integrations, but sampled source excerpts are incomplete, so links between unsampled subsystems are limited to those documented capabilities.
Main entry points: ray_trainer.py PPO trainer · engine_workers.py Engine workers · rollout Rollout workers · single_controller Worker controller
Components 23
Training workflows
- PPO entry
verl/trainer/main_ppo.py - PPO trainer
verl/trainer/ppo/ray_trainer.py - RL datasets
verl/utils/dataset/rl_dataset.py - Reward scoring
verl/workers/reward_manager - RL algorithms
verl/trainer/ppo/core_algos.py - SFT training
verl/trainer/sft_trainer.py - Evaluation
verl/trainer/main_eval.py
Distributed execution
- Rollout workers
verl/workers/rollout - Worker controller
verl/single_controller - Engine workers
verl/workers/engine_workers.py
Model and state
- Training engines
verl/workers/engine - Model adapters
verl/models - Checkpoint managers
verl/utils/checkpoint - Checkpoint transfer
verl/checkpoint_engine - Checkpoint storage
Post-training extensions
- Distillation
verl/trainer/distillation - Agent loops
verl/experimental/agent_loop - Async policy
verl/experimental/fully_async_policy - Reward loop
verl/experimental/reward_loop - Tool APIs
verl/tools
Other components
- Training user
- LLM frameworks
- Ray cluster
Connections 23
- Training user → PPO entry: configures run
- PPO entry → PPO trainer: starts training
- PPO trainer → RL datasets: loads prompts
- PPO trainer → Worker controller: dispatches work
- Worker controller → Engine workers: invokes workers
- Engine workers → Rollout workers: requests generation
- Rollout workers → Reward scoring: scores responses
- PPO trainer → RL algorithms: computes updates
- Engine workers → Training engines: runs computation
- Training engines → Model adapters: executes models
- PPO trainer → Checkpoint managers: saves state
- Checkpoint managers → Checkpoint storage: writes checkpoints
- Engine workers → Checkpoint transfer: syncs weights
- Checkpoint transfer → Rollout workers: updates weights
- Training engines → LLM frameworks: integrates backends
- Worker controller → Ray cluster: distributes workers
- Training user → SFT training: starts SFT
- Training user → Evaluation: starts evaluation
- Training user → Distillation: configures distillation
- PPO trainer → Async policy: supports async training
- Rollout workers → Agent loops: supports agent flows
- Agent loops → Tool APIs: invokes tools
- PPO trainer → Reward loop: supports reward workflows