How leaplabthu/opdvr works
OPDVR is a verl-based training system that combines teacher–student on-policy distillation with verifier-gated rewards, with an optional group-relative variant. The README documents the main training entry point, gated reward and GRPD logic, configuration, and in-loop validation; several implementation links below rely on that documentation because the relevant trainer and rollout sources were not sampled.
Main entry points: ray_trainer.py PPO Ray trainer · main_ppo.py PPO training entry · vllm_engine.py Teacher log-probabilities · actor Policy optimization
Components 16
Training inputs
- Training and validation data
- PPO training entry
verl/verl/trainer/main_ppo.py - OPDVR configuration
verl/verl/workers/config/rollout.py
Generation and verification
- PPO Ray trainer
verl/verl/trainer/ppo/ray_trainer.py - Student generation
verl/verl/workers/rollout - Teacher log-probabilities
verl/recipe/gkd/teacher/vllm_engine.py - Task verifier
verl/verl/trainer/ppo/reward.py
Distillation and learning
- Correctness-gated reward
verl/verl/trainer/ppo/ray_trainer.py - Group-relative scaling
verl/verl/trainer/ppo/ray_trainer.py - Policy optimization
verl/verl/workers/actor
Evaluation and persistence
- Model checkpoints
verl/verl/utils/checkpoint - In-loop validation
verl/verl/trainer/ppo/ray_trainer.py - Metrics reporting
verl/verl/utils/tracking.py
Other components
- Training operator
- Teacher model
- Wandb or console
Connections 22
- Training operator → PPO training entry: launches
- PPO training entry → OPDVR configuration: loads settings
- PPO training entry → PPO Ray trainer: starts training
- Training and validation data → PPO Ray trainer: supplies batches
- OPDVR configuration → PPO Ray trainer: configures
- PPO Ray trainer → Student generation: requests generations
- PPO Ray trainer → Teacher log-probabilities: requests log-probabilities
- Teacher model → Teacher log-probabilities: provides model
- PPO Ray trainer → Task verifier: requests task scores
- PPO Ray trainer → Correctness-gated reward: applies gated reward
- Task verifier → Correctness-gated reward: provides correctness
- Teacher log-probabilities → Correctness-gated reward: provides token guidance
- PPO Ray trainer → Group-relative scaling: optionally scales
- Correctness-gated reward → Group-relative scaling: provides gated signal
- PPO Ray trainer → Policy optimization: updates policy
- Correctness-gated reward → Policy optimization: provides reward signal
- Group-relative scaling → Policy optimization: provides scaled signal
- PPO Ray trainer → Model checkpoints: saves checkpoints
- PPO Ray trainer → In-loop validation: runs validation
- Training and validation data → In-loop validation: supplies validation set
- In-loop validation → Metrics reporting: reports metrics
- Metrics reporting → Wandb or console: emits results