# leaplabthu/opdvr architecture

> Architecture diagram of the GitHub repository leaplabthu/opdvr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/leaplabthu/opdvr
- Repository: https://github.com/leaplabthu/opdvr

## Overview

OPDVR is a verl-based training system that combines teacher–student on-policy distillation with verifier-gated rewards, with an optional group-relative variant. The README documents the main training entry point, gated reward and GRPD logic, configuration, and in-loop validation; several implementation links below rely on that documentation because the relevant trainer and rollout sources were not sampled.

Main entry points: [`verl/verl/trainer/ppo/ray_trainer.py`](https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/ppo/ray_trainer.py) (PPO Ray trainer), [`verl/verl/trainer/main_ppo.py`](https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/main_ppo.py) (PPO training entry), [`verl/recipe/gkd/teacher/vllm_engine.py`](https://github.com/leaplabthu/opdvr/blob/main/verl/recipe/gkd/teacher/vllm_engine.py) (Teacher log-probabilities), [`verl/verl/workers/actor`](https://github.com/leaplabthu/opdvr/tree/main/verl/verl/workers/actor) (Policy optimization)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_input["Training inputs"]
  node_data[("Training and validation data")]
  node_entry["PPO training entry<br/>[main_ppo.py]"]
  node_config["OPDVR configuration<br/>[rollout.py]"]
end

subgraph group_runtime["Generation and verification"]
  node_trainer["PPO Ray trainer<br/>[ray_trainer.py]"]
  node_rollout["Student generation"]
  node_teacher["Teacher log-probabilities<br/>[vllm_engine.py]"]
  node_verifier["Task verifier<br/>[reward.py]"]
end

subgraph group_learning["Distillation and learning"]
  node_gate["Correctness-gated reward<br/>[ray_trainer.py]"]
  node_advantage["Group-relative scaling<br/>[ray_trainer.py]"]
  node_actor["Policy optimization"]
end

subgraph group_results["Evaluation and persistence"]
  node_checkpoint["Model checkpoints"]
  node_validation["In-loop validation<br/>[ray_trainer.py]"]
  node_tracking["Metrics reporting<br/>[tracking.py]"]
end

node_operator(("Training operator"))
node_external_teacher(("Teacher model"))
node_external_tracking(("Wandb or console"))

node_operator -->|"launches"| node_entry
node_entry -->|"loads settings"| node_config
node_entry -->|"starts training"| node_trainer
node_data -->|"supplies batches"| node_trainer
node_config -->|"configures"| node_trainer
node_trainer -->|"requests generations"| node_rollout
node_trainer -->|"requests log-probabilities"| node_teacher
node_external_teacher -->|"provides model"| node_teacher
node_trainer -->|"requests task scores"| node_verifier
node_trainer -->|"applies gated reward"| node_gate
node_verifier -->|"provides correctness"| node_gate
node_teacher -->|"provides token guidance"| node_gate
node_trainer -.->|"optionally scales"| node_advantage
node_gate -.->|"provides gated signal"| node_advantage
node_trainer -->|"updates policy"| node_actor
node_gate -->|"provides reward signal"| node_actor
node_advantage -.->|"provides scaled signal"| node_actor
node_trainer -->|"saves checkpoints"| node_checkpoint
node_trainer -->|"runs validation"| node_validation
node_data -->|"supplies validation set"| node_validation
node_validation -->|"reports metrics"| node_tracking
node_tracking -->|"emits results"| node_external_tracking

click node_entry "https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/main_ppo.py"
click node_config "https://github.com/leaplabthu/opdvr/blob/main/verl/verl/workers/config/rollout.py"
click node_trainer "https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/ppo/ray_trainer.py"
click node_rollout "https://github.com/leaplabthu/opdvr/tree/main/verl/verl/workers/rollout"
click node_teacher "https://github.com/leaplabthu/opdvr/blob/main/verl/recipe/gkd/teacher/vllm_engine.py"
click node_verifier "https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/ppo/reward.py"
click node_gate "https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/ppo/ray_trainer.py"
click node_advantage "https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/ppo/ray_trainer.py"
click node_actor "https://github.com/leaplabthu/opdvr/tree/main/verl/verl/workers/actor"
click node_checkpoint "https://github.com/leaplabthu/opdvr/tree/main/verl/verl/utils/checkpoint"
click node_validation "https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/ppo/ray_trainer.py"
click node_tracking "https://github.com/leaplabthu/opdvr/blob/main/verl/verl/utils/tracking.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_data,node_entry,node_config toneBlue
class node_trainer,node_rollout,node_teacher,node_verifier toneAmber
class node_gate,node_advantage,node_actor toneMint
class node_checkpoint,node_validation,node_tracking toneRose
class node_operator,node_external_teacher,node_external_tracking toneIndigo
```

## Components

### Training inputs

- **Training and validation data**
- [PPO training entry](https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/main_ppo.py): `verl/verl/trainer/main_ppo.py`
- [OPDVR configuration](https://github.com/leaplabthu/opdvr/blob/main/verl/verl/workers/config/rollout.py): `verl/verl/workers/config/rollout.py`

### Generation and verification

- [PPO Ray trainer](https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/ppo/ray_trainer.py): `verl/verl/trainer/ppo/ray_trainer.py`
- [Student generation](https://github.com/leaplabthu/opdvr/tree/main/verl/verl/workers/rollout): `verl/verl/workers/rollout`
- [Teacher log-probabilities](https://github.com/leaplabthu/opdvr/blob/main/verl/recipe/gkd/teacher/vllm_engine.py): `verl/recipe/gkd/teacher/vllm_engine.py`
- [Task verifier](https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/ppo/reward.py): `verl/verl/trainer/ppo/reward.py`

### Distillation and learning

- [Correctness-gated reward](https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/ppo/ray_trainer.py): `verl/verl/trainer/ppo/ray_trainer.py`
- [Group-relative scaling](https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/ppo/ray_trainer.py): `verl/verl/trainer/ppo/ray_trainer.py`
- [Policy optimization](https://github.com/leaplabthu/opdvr/tree/main/verl/verl/workers/actor): `verl/verl/workers/actor`

### Evaluation and persistence

- [Model checkpoints](https://github.com/leaplabthu/opdvr/tree/main/verl/verl/utils/checkpoint): `verl/verl/utils/checkpoint`
- [In-loop validation](https://github.com/leaplabthu/opdvr/blob/main/verl/verl/trainer/ppo/ray_trainer.py): `verl/verl/trainer/ppo/ray_trainer.py`
- [Metrics reporting](https://github.com/leaplabthu/opdvr/blob/main/verl/verl/utils/tracking.py): `verl/verl/utils/tracking.py`

### Other components

- **Training operator**
- **Teacher model**
- **Wandb or console**

## Connections

- Training operator → PPO training entry: launches
- PPO training entry → OPDVR configuration: loads settings
- PPO training entry → PPO Ray trainer: starts training
- Training and validation data → PPO Ray trainer: supplies batches
- OPDVR configuration → PPO Ray trainer: configures
- PPO Ray trainer → Student generation: requests generations
- PPO Ray trainer → Teacher log-probabilities: requests log-probabilities
- Teacher model → Teacher log-probabilities: provides model
- PPO Ray trainer → Task verifier: requests task scores
- PPO Ray trainer → Correctness-gated reward: applies gated reward
- Task verifier → Correctness-gated reward: provides correctness
- Teacher log-probabilities → Correctness-gated reward: provides token guidance
- PPO Ray trainer → Group-relative scaling: optionally scales
- Correctness-gated reward → Group-relative scaling: provides gated signal
- PPO Ray trainer → Policy optimization: updates policy
- Correctness-gated reward → Policy optimization: provides reward signal
- Group-relative scaling → Policy optimization: provides scaled signal
- PPO Ray trainer → Model checkpoints: saves checkpoints
- PPO Ray trainer → In-loop validation: runs validation
- Training and validation data → In-loop validation: supplies validation set
- In-loop validation → Metrics reporting: reports metrics
- Metrics reporting → Wandb or console: emits results