How rsshyam/grpo works
Project Type and Purpose
• Project Type: A machine‐learning training pipeline for language models, specializing in two stages—Supervised Fine‐Tuning (SFT) and Group‐Robust Preference Optimization (GRIPO).
• Purpose: Orchestrate experiments (SFT, IPO, GRIPO) on configurable model–dataset combinations, log metrics to Weights & Biases (WandB), and support post-hoc plotting.
Main Architectural Components
a. Configuration Layer
• “config/” directory: YAML files for data‐selection, loss settings, model specs, and test experiments.
• Config Loader module (in src/utils.py or inside train.py): Parses config/config.yaml and merges with experiment‐specific files.
b. Data Layer
• Data Selection (src/data_selection.py): Chooses subsets or sampling strategies based on config.
• Preference Datasets (src/preference_datasets.py): Ingests raw preference data from the HuggingFace “llm_global_opinions” dataset; outputs PyTorch/TensorFlow Datasets.
• Data Processing (src/groupstuff/*.py): Implements group‐aware dataset splits for robust training.
c. Model Layer
• Model definitions (src/models.py): Model‐wrapper class to load Gemma‐2b (or other HF models).
• HuggingFace Model Repository (external service): Source of pretrained checkpoints.
d. Training Layer
• Trainer Factory (src/trainers_factory.py): Factory Pattern instantiates one of:
• BasicTrainer (SFT)
• ParallelTrainer (multi‐GPU IPO)
• GroupTrainer / GroupEarlyStopTrainer (GRIPO with group‐robust stopping)
• Loss Utilities (src/loss_utils.py): Implements DPO/IPO/GRIPO losses reading from config/loss/.
e. Evaluation & Logging
• Eval modules (src/eval/*.py): Win‐rate and fast OAI evaluations for post‐training validation.
• WandB service (external): Receives training and evaluation metrics.
f. Orchestration Scripts
• train.py: Main CLI entrypoint. Parses args, loads config, calls trainer factory, kicks off training.
• Shell scripts (scripts/*.sh): Preconfigured invocations of train.py for SFT (run_sft.sh), IPO (run_multi.sh), GRIPO (run_multi_robust.sh).
g. Plotting Layer
• plot_scripts/: Python scripts that fetch WandB metrics via API and generate both full and paper‐style plots.
Relationships & Data Flows
• Config Loader → Data Layer & Model Layer & Training Layer (decisions flow from config to each component).
• Data Layer → Training Layer (feeds training data batches).
• Model Layer + Loss Utilities → Training Layer (model + loss define optimization).
• Training Layer → WandB (logs metrics) & Checkpoint Storage (local/remote).
• Training Layer → Evaluation Modules (optionally invoked mid‐training or post‐training).
• Plotting Layer ← WandB API (reads logged metrics).
Architectural Patterns & Principles
• Factory Pattern in trainers_factory.py for extensible trainer selection.
• Config‐Driven Design: Almost every behavior is driven by YAML files.
• Separation of Concerns: Data ingestion, modeling, training, evaluation, and plotting live in distinct directories.