How vaibhav016/gdpo works
Project Type and Purpose
• This is a research‐oriented machine‐learning codebase implementing the “Group Diffusion Policy Optimization” (GDPO) algorithm on top of the D1 repository.
• It has three main pipelines:
• Supervised fine-tuning (SFT) of a diffusion language model
• Reinforcement-learning training (GDPO)
• Post‐training evaluation across reasoning/coding benchmarks
Top-Level Components (from file tree)
• dataset/
– Raw and preprocessed data: sudoku puzzles, mbpp coding problems, “countdown” math problems, etc.
• SFT/
– sft_train.py, sft_trainer.py, ddp_config.yaml – code + configs to run supervised fine-tuning via PyTorch DDP or Accelerate
• gdpo/
– gdpo_train.py, gdpo_trainer.py: the main RL loop implementing GDPO
– likelihood_estimators.py, reward_func.py: plug-in policy gradient components
– utils_code/: shared helper modules for data loading, logging (W&B), code execution clients, scoring
– slurm_scripts/: sample SLURM job definitions for GPU clusters
• eval/
– generate.py, eval.py: generation and high-level evaluation drivers
– parsers, parse_and_get_acc.py: benchmark-specific logic to compute accuracy
– run_eval.sh, run_eval_code.sh: helper wrappers to launch evaluations via torchrun
• main.py
– A top-level Python entrypoint orchestrating either SFT or GDPO based on CLI flags/config
• Configuration
– YAML files under SFT/, gdpo/accelerate.yaml and SLURM scripts drive distributed setup
– pyproject.toml, uv.lock: dependency & environment definitions
How Components Interact
• Data → SFT/GDPO pipelines
• Trainer classes in SFT and GDPO consume examples from dataset/
• Config files specify hyperparameters, model checkpoint paths, accelerator backends
• SFT → GDPO
• Optionally use an SFT-trained LoRA adapter or base model as starting checkpoint for RL
• GDPO → Checkpoint Store
• At intervals, gdpo_trainer serializes policy weights (LoRA adapters) to disk
• Evaluation → Checkpoint Store
• eval scripts load either base model or LoRA adapters, run generate.py to sample outputs
• parse_and_get_acc computes metrics per benchmark
• External Services
• SLURM schedules jobs (via scripts in gdpo/slurm_scripts/) on GPU clusters
• W&B logging from utils_code/wandb_logging.py
• HuggingFace Hub for model downloads (e.g., GSAI-ML/LLada-8B)
Architectural Patterns & Design Principles
• Modular pipeline separation: distinct folders for SFT, GDPO, evaluation, dataset
• Trainer abstraction: each pipeline implements a Trainer class encapsulating data loading, forward/backward passes, logging, checkpointing
• Config-driven: YAMLs and command-line flags drive behavior—no hardcoded hyperparameters
• Reusable utilities: gdpo/utils_code provides shared components (logging, data utils, code execution clients) to avoid duplication
• Batch scheduler-friendly: SLURM scripts are first-class citizens for large-scale runs