# thinking-space/rethinking-opd architecture

> Architecture diagram of the GitHub repository thinking-space/rethinking-opd, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thinking-space/rethinking-opd
- Repository: https://github.com/thinking-space/rethinking-opd

## Overview

This repository combines an OPD research workflow with verl for distributed on-policy training and LlamaFactory for supervised fine-tuning. The documented paths are: generate teacher responses for SFT, train the student; or collect student rollouts, obtain teacher token-level signals, and update the student, with separate evaluation. The graph retains the README’s top-k and recovery-strategy capabilities. Wiring for several training subsystems is inferred only from documented usage and the tree; source excerpts are partial, so uncertain internal calls are omitted.

Main entry points: [`verl/verl/trainer/main_ppo.py`](https://github.com/thinking-space/rethinking-opd/blob/main/verl/verl/trainer/main_ppo.py) (PPO training entry), [`verl/verl/trainer/ppo/ray_trainer.py`](https://github.com/thinking-space/rethinking-opd/blob/main/verl/verl/trainer/ppo/ray_trainer.py) (Student rollouts), [`verl/recipe/gkd/teacher/vllm_engine.py`](https://github.com/thinking-space/rethinking-opd/blob/main/verl/recipe/gkd/teacher/vllm_engine.py) (Teacher logprobs), [`scripts/infer/vllm_rollout.py`](https://github.com/thinking-space/rethinking-opd/blob/main/scripts/infer/vllm_rollout.py) (Teacher rollout)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sft["SFT preparation"]
  node_teacher_rollout["Teacher rollout<br/>[vllm_rollout.py]"]
  node_sft_data["SFT responses<br/>[loader.py]"]
  node_sft_train["SFT training<br/>[workflow.py]"]
  node_lf_data["Data processing<br/>[supervised.py]"]
end

subgraph group_opd["OPD training"]
  node_opd_entry["PPO training entry<br/>[main_ppo.py]"]
  node_rollouts["Student rollouts<br/>[ray_trainer.py]"]
  node_reward["Token reward<br/>[reward.py]"]
  node_teacher_engine["Teacher logprobs<br/>[vllm_engine.py]"]
  node_opd_update["Policy update<br/>[core_algos.py]"]
  node_checkpoint["Checkpointing"]
  node_topk["Top-k strategies"]
  node_recovery["OPD recovery"]
end

subgraph group_models["Model runtime"]
  node_model_loader["Model loading<br/>[loader.py]"]
  node_infer_runtime["Inference engines"]
end

subgraph group_evaluation["Evaluation"]
  node_evaluation["Checkpoint evaluation"]
  node_metrics["Scoring<br/>[grade.py]"]
  node_results["Evaluation results"]
end

node_researcher(("Researcher"))
node_prompt_data["Prompt data"]
node_teacher["Teacher model"]
node_student["Student model"]

node_researcher -->|"starts rollout"| node_teacher_rollout
node_prompt_data -->|"provides prompts"| node_teacher_rollout
node_teacher -->|"generates responses"| node_teacher_rollout
node_teacher_rollout -->|"writes responses"| node_sft_data
node_sft_data -->|"loads examples"| node_lf_data
node_lf_data -->|"prepares batches"| node_sft_train
node_sft_train -->|"trains student"| node_student
node_researcher -->|"starts training"| node_opd_entry
node_opd_entry -->|"coordinates generation"| node_rollouts
node_student -->|"generates samples"| node_rollouts
node_rollouts -->|"supplies samples"| node_reward
node_opd_entry -->|"requests teacher signals"| node_teacher_engine
node_teacher -->|"provides logprobs"| node_teacher_engine
node_teacher_engine -->|"supplies logprobs"| node_reward
node_rollouts -->|"selects token set"| node_topk
node_teacher_engine -->|"provides token scores"| node_topk
node_topk -->|"configures token rewards"| node_reward
node_reward -->|"provides learning signal"| node_opd_update
node_opd_update -->|"updates policy"| node_student
node_opd_entry -->|"saves training state"| node_checkpoint
node_researcher -->|"prepares recovery data"| node_recovery
node_recovery -->|"supports cold-start SFT"| node_lf_data
node_student -->|"provides checkpoint"| node_evaluation
node_evaluation -->|"scores outputs"| node_metrics
node_metrics -->|"writes scores"| node_results
node_model_loader -->|"loads model"| node_student
node_model_loader -->|"loads model"| node_teacher

click node_teacher_rollout "https://github.com/thinking-space/rethinking-opd/blob/main/scripts/infer/vllm_rollout.py"
click node_sft_data "https://github.com/thinking-space/rethinking-opd/blob/main/LlamaFactory/src/llamafactory/data/loader.py"
click node_sft_train "https://github.com/thinking-space/rethinking-opd/blob/main/LlamaFactory/src/llamafactory/train/sft/workflow.py"
click node_lf_data "https://github.com/thinking-space/rethinking-opd/blob/main/LlamaFactory/src/llamafactory/data/processor/supervised.py"
click node_opd_entry "https://github.com/thinking-space/rethinking-opd/blob/main/verl/verl/trainer/main_ppo.py"
click node_rollouts "https://github.com/thinking-space/rethinking-opd/blob/main/verl/verl/trainer/ppo/ray_trainer.py"
click node_reward "https://github.com/thinking-space/rethinking-opd/blob/main/verl/verl/trainer/ppo/reward.py"
click node_teacher_engine "https://github.com/thinking-space/rethinking-opd/blob/main/verl/recipe/gkd/teacher/vllm_engine.py"
click node_opd_update "https://github.com/thinking-space/rethinking-opd/blob/main/verl/verl/trainer/ppo/core_algos.py"
click node_checkpoint "https://github.com/thinking-space/rethinking-opd/blob/main/verl/verl/utils/checkpoint/checkpoint_manager.py"
click node_topk "https://github.com/thinking-space/rethinking-opd/tree/main/verl/verl/trainer/ppo"
click node_recovery "https://github.com/thinking-space/rethinking-opd/tree/main/scripts/infer"
click node_model_loader "https://github.com/thinking-space/rethinking-opd/blob/main/LlamaFactory/src/llamafactory/model/loader.py"
click node_infer_runtime "https://github.com/thinking-space/rethinking-opd/tree/main/LlamaFactory/src/llamafactory/chat"
click node_evaluation "https://github.com/thinking-space/rethinking-opd/tree/main/scripts/val/eval"
click node_metrics "https://github.com/thinking-space/rethinking-opd/blob/main/scripts/val/eval/grade.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_teacher_rollout,node_sft_data,node_sft_train,node_lf_data toneBlue
class node_opd_entry,node_rollouts,node_reward,node_teacher_engine,node_opd_update,node_checkpoint,node_topk,node_recovery toneAmber
class node_model_loader,node_infer_runtime toneMint
class node_evaluation,node_metrics,node_results toneRose
class node_researcher,node_prompt_data,node_teacher,node_student toneIndigo
```

## Components

### SFT preparation

- [Teacher rollout](https://github.com/thinking-space/rethinking-opd/blob/main/scripts/infer/vllm_rollout.py): `scripts/infer/vllm_rollout.py`
- [SFT responses](https://github.com/thinking-space/rethinking-opd/blob/main/LlamaFactory/src/llamafactory/data/loader.py): `LlamaFactory/src/llamafactory/data/loader.py`
- [SFT training](https://github.com/thinking-space/rethinking-opd/blob/main/LlamaFactory/src/llamafactory/train/sft/workflow.py): `LlamaFactory/src/llamafactory/train/sft/workflow.py`
- [Data processing](https://github.com/thinking-space/rethinking-opd/blob/main/LlamaFactory/src/llamafactory/data/processor/supervised.py): `LlamaFactory/src/llamafactory/data/processor/supervised.py`

### OPD training

- [PPO training entry](https://github.com/thinking-space/rethinking-opd/blob/main/verl/verl/trainer/main_ppo.py): `verl/verl/trainer/main_ppo.py`
- [Student rollouts](https://github.com/thinking-space/rethinking-opd/blob/main/verl/verl/trainer/ppo/ray_trainer.py): `verl/verl/trainer/ppo/ray_trainer.py`
- [Token reward](https://github.com/thinking-space/rethinking-opd/blob/main/verl/verl/trainer/ppo/reward.py): `verl/verl/trainer/ppo/reward.py`
- [Teacher logprobs](https://github.com/thinking-space/rethinking-opd/blob/main/verl/recipe/gkd/teacher/vllm_engine.py): `verl/recipe/gkd/teacher/vllm_engine.py`
- [Policy update](https://github.com/thinking-space/rethinking-opd/blob/main/verl/verl/trainer/ppo/core_algos.py): `verl/verl/trainer/ppo/core_algos.py`
- [Checkpointing](https://github.com/thinking-space/rethinking-opd/blob/main/verl/verl/utils/checkpoint/checkpoint_manager.py): `verl/verl/utils/checkpoint/checkpoint_manager.py`
- [Top-k strategies](https://github.com/thinking-space/rethinking-opd/tree/main/verl/verl/trainer/ppo): `verl/verl/trainer/ppo`
- [OPD recovery](https://github.com/thinking-space/rethinking-opd/tree/main/scripts/infer): `scripts/infer`

### Model runtime

- [Model loading](https://github.com/thinking-space/rethinking-opd/blob/main/LlamaFactory/src/llamafactory/model/loader.py): `LlamaFactory/src/llamafactory/model/loader.py`
- [Inference engines](https://github.com/thinking-space/rethinking-opd/tree/main/LlamaFactory/src/llamafactory/chat): `LlamaFactory/src/llamafactory/chat`

### Evaluation

- [Checkpoint evaluation](https://github.com/thinking-space/rethinking-opd/tree/main/scripts/val/eval): `scripts/val/eval`
- [Scoring](https://github.com/thinking-space/rethinking-opd/blob/main/scripts/val/eval/grade.py): `scripts/val/eval/grade.py`
- **Evaluation results**

### Other components

- **Researcher**
- **Prompt data**
- **Teacher model**
- **Student model**

## Connections

- Researcher → Teacher rollout: starts rollout
- Prompt data → Teacher rollout: provides prompts
- Teacher model → Teacher rollout: generates responses
- Teacher rollout → SFT responses: writes responses
- SFT responses → Data processing: loads examples
- Data processing → SFT training: prepares batches
- SFT training → Student model: trains student
- Researcher → PPO training entry: starts training
- PPO training entry → Student rollouts: coordinates generation
- Student model → Student rollouts: generates samples
- Student rollouts → Token reward: supplies samples
- PPO training entry → Teacher logprobs: requests teacher signals
- Teacher model → Teacher logprobs: provides logprobs
- Teacher logprobs → Token reward: supplies logprobs
- Student rollouts → Top-k strategies: selects token set
- Teacher logprobs → Top-k strategies: provides token scores
- Top-k strategies → Token reward: configures token rewards
- Token reward → Policy update: provides learning signal
- Policy update → Student model: updates policy
- PPO training entry → Checkpointing: saves training state
- Researcher → OPD recovery: prepares recovery data
- OPD recovery → Data processing: supports cold-start SFT
- Student model → Checkpoint evaluation: provides checkpoint
- Checkpoint evaluation → Scoring: scores outputs
- Scoring → Evaluation results: writes scores
- Model loading → Student model: loads model
- Model loading → Teacher model: loads model