How thinking-space/rethinking-opd works
This repository combines an OPD research workflow with verl for distributed on-policy training and LlamaFactory for supervised fine-tuning. The documented paths are: generate teacher responses for SFT, train the student; or collect student rollouts, obtain teacher token-level signals, and update the student, with separate evaluation. The graph retains the README’s top-k and recovery-strategy capabilities. Wiring for several training subsystems is inferred only from documented usage and the tree; source excerpts are partial, so uncertain internal calls are omitted.
Main entry points: main_ppo.py PPO training entry · ray_trainer.py Student rollouts · vllm_engine.py Teacher logprobs · vllm_rollout.py Teacher rollout
Components 21
SFT preparation
- Teacher rollout
scripts/infer/vllm_rollout.py - SFT responses
LlamaFactory/src/llamafactory/data/loader.py - SFT training
LlamaFactory/src/llamafactory/train/sft/workflow.py - Data processing
LlamaFactory/src/llamafactory/data/processor/supervised.py
OPD training
- PPO training entry
verl/verl/trainer/main_ppo.py - Student rollouts
verl/verl/trainer/ppo/ray_trainer.py - Token reward
verl/verl/trainer/ppo/reward.py - Teacher logprobs
verl/recipe/gkd/teacher/vllm_engine.py - Policy update
verl/verl/trainer/ppo/core_algos.py - Checkpointing
verl/verl/utils/checkpoint/checkpoint_manager.py - Top-k strategies
verl/verl/trainer/ppo - OPD recovery
scripts/infer
Model runtime
- Model loading
LlamaFactory/src/llamafactory/model/loader.py - Inference engines
LlamaFactory/src/llamafactory/chat
Evaluation
- Checkpoint evaluation
scripts/val/eval - Scoring
scripts/val/eval/grade.py - Evaluation results
Other components
- Researcher
- Prompt data
- Teacher model
- Student model
Connections 27
- Researcher → Teacher rollout: starts rollout
- Prompt data → Teacher rollout: provides prompts
- Teacher model → Teacher rollout: generates responses
- Teacher rollout → SFT responses: writes responses
- SFT responses → Data processing: loads examples
- Data processing → SFT training: prepares batches
- SFT training → Student model: trains student
- Researcher → PPO training entry: starts training
- PPO training entry → Student rollouts: coordinates generation
- Student model → Student rollouts: generates samples
- Student rollouts → Token reward: supplies samples
- PPO training entry → Teacher logprobs: requests teacher signals
- Teacher model → Teacher logprobs: provides logprobs
- Teacher logprobs → Token reward: supplies logprobs
- Student rollouts → Top-k strategies: selects token set
- Teacher logprobs → Top-k strategies: provides token scores
- Top-k strategies → Token reward: configures token rewards
- Token reward → Policy update: provides learning signal
- Policy update → Student model: updates policy
- PPO training entry → Checkpointing: saves training state
- Researcher → OPD recovery: prepares recovery data
- OPD recovery → Data processing: supports cold-start SFT
- Student model → Checkpoint evaluation: provides checkpoint
- Checkpoint evaluation → Scoring: scores outputs
- Scoring → Evaluation results: writes scores
- Model loading → Student model: loads model
- Model loading → Teacher model: loads model