# noncollapse/demystifying-grpo architecture

> Architecture diagram of the GitHub repository noncollapse/demystifying-grpo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/noncollapse/demystifying-grpo
- Repository: https://github.com/noncollapse/demystifying-grpo

## Overview

This repository reproduces GRPO gradient-estimator experiments and provides a separate GRPO policy-training launcher. The main evaluation workflow computes per-prompt Monte Carlo oracle values, then uses those values with sampled model continuations to compare gradient-estimator covariance traces. The README documents plotting and VERL-based training as additional capabilities. The training launcher’s internals are not sampled, so its connections are limited to documented behavior.

Main entry points: [`Gradient_evaluation/run_tracecov.py`](https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/run_tracecov.py) (Trace-Covariance Run), [`Gradient_evaluation/calculate_oracle_v.py`](https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/calculate_oracle_v.py) (Oracle Computation), [`Policy_Optimization/grpo_train.sh`](https://github.com/noncollapse/demystifying-grpo/blob/main/Policy_Optimization/grpo_train.sh) (GRPO Launcher)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_oracle["Oracle Values"]
  node_oracle_job["Oracle Computation"]
  node_prompt_format["Prompt Formatting"]
  node_oracle_rollouts["Monte Carlo Rollouts"]
  node_reward["Answer Reward"]
  node_oracle_results["Oracle Results"]
end

subgraph group_evaluation["Gradient Evaluation"]
  node_trace_job["Trace-Covariance Run<br/>[run_tracecov.py]"]
  node_table_loader["Oracle Table Loader<br/>[run_tracecov.py]"]
  node_sample["Continuation Sampling<br/>[run_tracecov.py]"]
  node_estimator["Gradient Estimators<br/>[run_tracecov.py]"]
  node_covariance["Covariance Statistics<br/>[run_tracecov.py]"]
  node_eval_results["Evaluation Results"]
  node_plots["Result Plots"]
end

subgraph group_training["Policy Training"]
  node_train_launcher["GRPO Launcher<br/>[grpo_train.sh]"]
end

node_researcher(("Researcher"))
node_dataset[("Hugging Face Dataset")]
node_model["Causal Language Model"]
node_verl["VERL Trainer"]
node_parquet[("Parquet Training Data")]
node_wandb["Weights &amp; Biases"]

node_researcher -->|"configures and runs"| node_oracle_job
node_oracle_job -->|"loads split"| node_dataset
node_oracle_job -->|"loads model"| node_model
node_oracle_job -->|"formats prompts"| node_prompt_format
node_oracle_job -->|"computes values with"| node_oracle_rollouts
node_oracle_rollouts -->|"generates continuations"| node_model
node_oracle_rollouts -->|"scores continuations"| node_reward
node_oracle_job -->|"writes JSONL"| node_oracle_results
node_researcher -->|"configures and runs"| node_trace_job
node_trace_job -->|"loads oracle table"| node_table_loader
node_table_loader -->|"reads"| node_oracle_results
node_trace_job -->|"loads model"| node_model
node_trace_job -->|"requests samples"| node_sample
node_sample -->|"generates continuations"| node_model
node_trace_job -->|"computes"| node_estimator
node_estimator -->|"scores answers"| node_reward
node_trace_job -->|"accumulates statistics"| node_covariance
node_trace_job -->|"saves metrics"| node_eval_results
node_trace_job -->|"produces plots"| node_plots
node_researcher -->|"configures and runs"| node_train_launcher
node_train_launcher -->|"uses configured data"| node_parquet
node_train_launcher -->|"launches training"| node_verl
node_train_launcher -.->|"logs metrics"| node_wandb

click node_oracle_job "https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/calculate_oracle_v.py"
click node_prompt_format "https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/calculate_oracle_v.py"
click node_oracle_rollouts "https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/calculate_oracle_v.py"
click node_reward "https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/calculate_oracle_v.py"
click node_trace_job "https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/run_tracecov.py"
click node_table_loader "https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/run_tracecov.py"
click node_sample "https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/run_tracecov.py"
click node_estimator "https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/run_tracecov.py"
click node_covariance "https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/run_tracecov.py"
click node_train_launcher "https://github.com/noncollapse/demystifying-grpo/blob/main/Policy_Optimization/grpo_train.sh"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_oracle_job,node_prompt_format,node_oracle_rollouts,node_reward,node_oracle_results toneBlue
class node_trace_job,node_table_loader,node_sample,node_estimator,node_covariance,node_eval_results,node_plots,node_dataset,node_parquet toneAmber
class node_train_launcher toneMint
class node_researcher,node_model,node_verl,node_wandb toneIndigo
```

## Components

### Oracle Values

- [Oracle Computation](https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/calculate_oracle_v.py): `Gradient_evaluation/calculate_oracle_v.py`
- [Prompt Formatting](https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/calculate_oracle_v.py): `Gradient_evaluation/calculate_oracle_v.py`
- [Monte Carlo Rollouts](https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/calculate_oracle_v.py): `Gradient_evaluation/calculate_oracle_v.py`
- [Answer Reward](https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/calculate_oracle_v.py): `Gradient_evaluation/calculate_oracle_v.py`
- **Oracle Results**

### Gradient Evaluation

- [Trace-Covariance Run](https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/run_tracecov.py): `Gradient_evaluation/run_tracecov.py`
- [Oracle Table Loader](https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/run_tracecov.py): `Gradient_evaluation/run_tracecov.py`
- [Continuation Sampling](https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/run_tracecov.py): `Gradient_evaluation/run_tracecov.py`
- [Gradient Estimators](https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/run_tracecov.py): `Gradient_evaluation/run_tracecov.py`
- [Covariance Statistics](https://github.com/noncollapse/demystifying-grpo/blob/main/Gradient_evaluation/run_tracecov.py): `Gradient_evaluation/run_tracecov.py`
- **Evaluation Results**
- **Result Plots**

### Policy Training

- [GRPO Launcher](https://github.com/noncollapse/demystifying-grpo/blob/main/Policy_Optimization/grpo_train.sh): `Policy_Optimization/grpo_train.sh`

### Other components

- **Researcher**
- **Hugging Face Dataset**
- **Causal Language Model**
- **VERL Trainer**
- **Parquet Training Data**
- **Weights & Biases**

## Connections

- Researcher → Oracle Computation: configures and runs
- Oracle Computation → Hugging Face Dataset: loads split
- Oracle Computation → Causal Language Model: loads model
- Oracle Computation → Prompt Formatting: formats prompts
- Oracle Computation → Monte Carlo Rollouts: computes values with
- Monte Carlo Rollouts → Causal Language Model: generates continuations
- Monte Carlo Rollouts → Answer Reward: scores continuations
- Oracle Computation → Oracle Results: writes JSONL
- Researcher → Trace-Covariance Run: configures and runs
- Trace-Covariance Run → Oracle Table Loader: loads oracle table
- Oracle Table Loader → Oracle Results: reads
- Trace-Covariance Run → Causal Language Model: loads model
- Trace-Covariance Run → Continuation Sampling: requests samples
- Continuation Sampling → Causal Language Model: generates continuations
- Trace-Covariance Run → Gradient Estimators: computes
- Gradient Estimators → Answer Reward: scores answers
- Trace-Covariance Run → Covariance Statistics: accumulates statistics
- Trace-Covariance Run → Evaluation Results: saves metrics
- Trace-Covariance Run → Result Plots: produces plots
- Researcher → GRPO Launcher: configures and runs
- GRPO Launcher → Parquet Training Data: uses configured data
- GRPO Launcher → VERL Trainer: launches training
- GRPO Launcher → Weights & Biases: logs metrics