# vaibhav016/gdpo architecture

> Architecture diagram of the GitHub repository vaibhav016/gdpo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vaibhav016/gdpo
- Repository: https://github.com/vaibhav016/gdpo

## Overview

##### Project Type and Purpose
- This is a research‐oriented machine‐learning codebase implementing the “Group Diffusion Policy Optimization” (GDPO) algorithm on top of the D1 repository.
- It has three main pipelines:
- Supervised fine-tuning (SFT) of a diffusion language model
- Reinforcement-learning training (GDPO)
- Post‐training evaluation across reasoning/coding benchmarks

##### Top-Level Components (from file tree)
- dataset/
– Raw and preprocessed data: sudoku puzzles, mbpp coding problems, “countdown” math problems, etc.
- SFT/
– sft_train.py, sft_trainer.py, ddp_config.yaml – code + configs to run supervised fine-tuning via PyTorch DDP or Accelerate
- gdpo/
– gdpo_train.py, gdpo_trainer.py: the main RL loop implementing GDPO
– likelihood_estimators.py, reward_func.py: plug-in policy gradient components
– utils_code/: shared helper modules for data loading, logging (W&B), code execution clients, scoring
– slurm_scripts/: sample SLURM job definitions for GPU clusters
- eval/
– generate.py, eval.py: generation and high-level evaluation drivers
– parsers, parse_and_get_acc.py: benchmark-specific logic to compute accuracy
– run_eval.sh, run_eval_code.sh: helper wrappers to launch evaluations via torchrun
- main.py
– A top-level Python entrypoint orchestrating either SFT or GDPO based on CLI flags/config
- Configuration
– YAML files under SFT/, gdpo/accelerate.yaml and SLURM scripts drive distributed setup
– pyproject.toml, uv.lock: dependency & environment definitions

##### How Components Interact
- Data → SFT/GDPO pipelines
- Trainer classes in SFT and GDPO consume examples from dataset/
- Config files specify hyperparameters, model checkpoint paths, accelerator backends
- SFT → GDPO
- Optionally use an SFT-trained LoRA adapter or base model as starting checkpoint for RL
- GDPO → Checkpoint Store
- At intervals, gdpo_trainer serializes policy weights (LoRA adapters) to disk
- Evaluation → Checkpoint Store
- eval scripts load either base model or LoRA adapters, run generate.py to sample outputs
- parse_and_get_acc computes metrics per benchmark
- External Services
- SLURM schedules jobs (via scripts in gdpo/slurm_scripts/) on GPU clusters
- W&B logging from utils_code/wandb_logging.py
- HuggingFace Hub for model downloads (e.g., GSAI-ML/LLada-8B)

##### Architectural Patterns & Design Principles
- Modular pipeline separation: distinct folders for SFT, GDPO, evaluation, dataset
- Trainer abstraction: each pipeline implements a Trainer class encapsulating data loading, forward/backward passes, logging, checkpointing
- Config-driven: YAMLs and command-line flags drive behavior—no hardcoded hyperparameters
- Reusable utilities: gdpo/utils_code provides shared components (logging, data utils, code execution clients) to avoid duplication
- Batch scheduler-friendly: SLURM scripts are first-class citizens for large-scale runs

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer" 
        direction TB
        SudokuPuzzle["4x4_sudoku_unique_puzzles.csv"]:::data
        SudokuTest["4x4_test_sudoku.csv"]:::data
        Countdown["countdown_cd3_test.jsonl"]:::data
        MBPPData["mbpp/mbpp.jsonl"]:::data
    end

    %% SFT Pipeline
    subgraph "Supervised Fine-Tuning (SFT)" 
        direction TB
        SFT_Train["sft_train.py"]:::train
        SFT_Trainer["sft_trainer.py"]:::train
        SFT_Config["ddp_config.yaml"]:::config
    end

    %% GDPO Pipeline
    subgraph "GDPO Pipeline" 
        direction TB
        GDPO_Train["gdpo_train.py"]:::train
        GDPO_Trainer["gdpo_trainer.py"]:::train
        Likelihood["likelihood_estimators.py"]:::train
        Reward["reward_func.py"]:::train
        AccelerateCfg["accelerate.yaml"]:::config
        subgraph "utils_code/" 
            direction TB
            UtilsCode["utils_code"]:::train
        end
        subgraph "slurm_scripts/" 
            direction TB
            SlurmAccel["accelerate_a100.yaml"]:::config
            SlurmCountdown["countdown_base.sbatch"]:::config
            SlurmTrain["train.yaml"]:::config
        end
    end

    %% Checkpoint Store
    CS[(Checkpoint Store)]:::data

    %% Evaluation Pipeline
    subgraph "Evaluation" 
        direction TB
        EvalGen["generate.py"]:::eval
        EvalRun["eval.py"]:::eval
        ParseAcc["parse_and_get_acc.py"]:::eval
        RunEval["run_eval.sh"]:::eval
        RunEvalCode["run_eval_code.sh"]:::eval
    end

    %% Control Plane
    subgraph "Control Plane" 
        direction TB
        Main["main.py"]:::config
        Pyproj["pyproject.toml"]:::config
        UV["uv.lock"]:::config
    end

    %% External Services
    SS[/"SLURM Cluster"/]:::external
    WB[(W&#38;B Logging)]:::external
    HF[(HuggingFace Hub)]:::external

    %% Data flows
    SudokuPuzzle -->|loads| SFT_Train
    SudokuTest -->|loads| SFT_Train
    Countdown -->|loads| SFT_Train
    MBPPData -->|loads| SFT_Train

    SudokuPuzzle -->|loads| GDPO_Trainer
    SudokuTest -->|loads| GDPO_Trainer
    Countdown -->|loads| GDPO_Trainer
    MBPPData -->|loads| GDPO_Trainer

    %% Pipeline flows
    SFT_Train -->|trains| SFT_Trainer
    SFT_Trainer -->|saves checkpoint| CS
    CS -->|initial checkpoint| GDPO_Trainer

    GDPO_Train -->|runs| GDPO_Trainer
    GDPO_Trainer -->|saves checkpoint| CS
    CS -->|loads| EvalGen

    EvalGen -->|generates| EvalRun
    EvalRun -->|reports| ParseAcc

    %% Control plane arrows
    Main --> SFT_Train
    Main --> GDPO_Train
    Main --> EvalGen

    Pyproj --> Main
    UV --> Main
    SFT_Config --> SFT_Train
    AccelerateCfg --> GDPO_Trainer
    SlurmAccel --> GDPO_Train
    SlurmCountdown --> GDPO_Trainer
    SlurmTrain --> GDPO_Trainer

    %% External integrations
    SS --> SFT_Train
    SS --> GDPO_Trainer
    SS --> EvalGen

    SFT_Trainer --> WB
    GDPO_Trainer --> WB
    ParseAcc --> WB

    SFT_Train --> HF
    GDPO_Train --> HF
    EvalGen --> HF

    %% Click Events
    click SudokuPuzzle "https://github.com/vaibhav016/gdpo/blob/main/dataset/4x4_sudoku_unique_puzzles.csv"
    click SudokuTest "https://github.com/vaibhav016/gdpo/blob/main/dataset/4x4_test_sudoku.csv"
    click Countdown "https://github.com/vaibhav016/gdpo/blob/main/dataset/countdown_cd3_test.jsonl"
    click MBPPData "https://github.com/vaibhav016/gdpo/blob/main/dataset/mbpp/mbpp.jsonl"

    click SFT_Train "https://github.com/vaibhav016/gdpo/blob/main/SFT/sft_train.py"
    click SFT_Trainer "https://github.com/vaibhav016/gdpo/blob/main/SFT/sft_trainer.py"
    click SFT_Config "https://github.com/vaibhav016/gdpo/blob/main/SFT/ddp_config.yaml"

    click GDPO_Train "https://github.com/vaibhav016/gdpo/blob/main/gdpo/gdpo_train.py"
    click GDPO_Trainer "https://github.com/vaibhav016/gdpo/blob/main/gdpo/gdpo_trainer.py"
    click Likelihood "https://github.com/vaibhav016/gdpo/blob/main/gdpo/likelihood_estimators.py"
    click Reward "https://github.com/vaibhav016/gdpo/blob/main/gdpo/reward_func.py"
    click AccelerateCfg "https://github.com/vaibhav016/gdpo/blob/main/gdpo/accelerate.yaml"
    click UtilsCode "https://github.com/vaibhav016/gdpo/tree/main/gdpo/utils_code/"
    click SlurmAccel "https://github.com/vaibhav016/gdpo/blob/main/gdpo/slurm_scripts/accelerate_a100.yaml"
    click SlurmCountdown "https://github.com/vaibhav016/gdpo/blob/main/gdpo/slurm_scripts/countdown_base.sbatch"
    click SlurmTrain "https://github.com/vaibhav016/gdpo/blob/main/gdpo/slurm_scripts/train.yaml"

    click EvalGen "https://github.com/vaibhav016/gdpo/blob/main/eval/generate.py"
    click EvalRun "https://github.com/vaibhav016/gdpo/blob/main/eval/eval.py"
    click ParseAcc "https://github.com/vaibhav016/gdpo/blob/main/eval/parse_and_get_acc.py"
    click RunEval "https://github.com/vaibhav016/gdpo/blob/main/eval/run_eval.sh"
    click RunEvalCode "https://github.com/vaibhav016/gdpo/blob/main/eval/run_eval_code.sh"

    click Main "https://github.com/vaibhav016/gdpo/blob/main/main.py"
    click Pyproj "https://github.com/vaibhav016/gdpo/blob/main/pyproject.toml"
    click UV "https://github.com/vaibhav016/gdpo/blob/main/uv.lock"

    %% Styles
    classDef data fill:#e0ffe0,stroke:#2d6a2d
    classDef train fill:#e0f0ff,stroke:#28527a
    classDef eval fill:#ffe0b3,stroke:#b36b00
    classDef config fill:#f0f0f0,stroke:#666666
    classDef external fill:#ffcccc,stroke:#a00000
```
