How noncollapse/demystifying-grpo works
This repository reproduces GRPO gradient-estimator experiments and provides a separate GRPO policy-training launcher. The main evaluation workflow computes per-prompt Monte Carlo oracle values, then uses those values with sampled model continuations to compare gradient-estimator covariance traces. The README documents plotting and VERL-based training as additional capabilities. The training launcher’s internals are not sampled, so its connections are limited to documented behavior.
Main entry points: run_tracecov.py Trace-Covariance Run · calculate_oracle_v.py Oracle Computation · grpo_train.sh GRPO Launcher
Components 19
Oracle Values
- Oracle Computation
Gradient_evaluation/calculate_oracle_v.py - Prompt Formatting
Gradient_evaluation/calculate_oracle_v.py - Monte Carlo Rollouts
Gradient_evaluation/calculate_oracle_v.py - Answer Reward
Gradient_evaluation/calculate_oracle_v.py - Oracle Results
Gradient Evaluation
- Trace-Covariance Run
Gradient_evaluation/run_tracecov.py - Oracle Table Loader
Gradient_evaluation/run_tracecov.py - Continuation Sampling
Gradient_evaluation/run_tracecov.py - Gradient Estimators
Gradient_evaluation/run_tracecov.py - Covariance Statistics
Gradient_evaluation/run_tracecov.py - Evaluation Results
- Result Plots
Policy Training
- GRPO Launcher
Policy_Optimization/grpo_train.sh
Other components
- Researcher
- Hugging Face Dataset
- Causal Language Model
- VERL Trainer
- Parquet Training Data
- Weights & Biases
Connections 23
- Researcher → Oracle Computation: configures and runs
- Oracle Computation → Hugging Face Dataset: loads split
- Oracle Computation → Causal Language Model: loads model
- Oracle Computation → Prompt Formatting: formats prompts
- Oracle Computation → Monte Carlo Rollouts: computes values with
- Monte Carlo Rollouts → Causal Language Model: generates continuations
- Monte Carlo Rollouts → Answer Reward: scores continuations
- Oracle Computation → Oracle Results: writes JSONL
- Researcher → Trace-Covariance Run: configures and runs
- Trace-Covariance Run → Oracle Table Loader: loads oracle table
- Oracle Table Loader → Oracle Results: reads
- Trace-Covariance Run → Causal Language Model: loads model
- Trace-Covariance Run → Continuation Sampling: requests samples
- Continuation Sampling → Causal Language Model: generates continuations
- Trace-Covariance Run → Gradient Estimators: computes
- Gradient Estimators → Answer Reward: scores answers
- Trace-Covariance Run → Covariance Statistics: accumulates statistics
- Trace-Covariance Run → Evaluation Results: saves metrics
- Trace-Covariance Run → Result Plots: produces plots
- Researcher → GRPO Launcher: configures and runs
- GRPO Launcher → Parquet Training Data: uses configured data
- GRPO Launcher → VERL Trainer: launches training
- GRPO Launcher → Weights & Biases: logs metrics