# rsshyam/grpo architecture

> Architecture diagram of the GitHub repository rsshyam/grpo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rsshyam/grpo
- Repository: https://github.com/rsshyam/grpo

## Overview

##### Project Type and Purpose
- Project Type: A machine‐learning training pipeline for language models, specializing in two stages—Supervised Fine‐Tuning (SFT) and Group‐Robust Preference Optimization (GRIPO).
- Purpose: Orchestrate experiments (SFT, IPO, GRIPO) on configurable model–dataset combinations, log metrics to Weights & Biases (WandB), and support post-hoc plotting.

##### Main Architectural Components
a. Configuration Layer
- “config/” directory: YAML files for data‐selection, loss settings, model specs, and test experiments.
- Config Loader module (in `src/utils.py` or inside `train.py`): Parses `config/config.yaml` and merges with experiment‐specific files.

b. Data Layer
- Data Selection (`src/data_selection.py`): Chooses subsets or sampling strategies based on config.
- Preference Datasets (`src/preference_datasets.py`): Ingests raw preference data from the HuggingFace “llm_global_opinions” dataset; outputs PyTorch/TensorFlow Datasets.
- Data Processing (`src/groupstuff/*.py`): Implements group‐aware dataset splits for robust training.

c. Model Layer
- Model definitions (`src/models.py`): Model‐wrapper class to load Gemma‐2b (or other HF models).
- HuggingFace Model Repository (external service): Source of pretrained checkpoints.

d. Training Layer
- Trainer Factory (`src/trainers_factory.py`): Factory Pattern instantiates one of:
- `BasicTrainer` (SFT)
- `ParallelTrainer` (multi‐GPU IPO)
- `GroupTrainer` / `GroupEarlyStopTrainer` (GRIPO with group‐robust stopping)
- Loss Utilities (`src/loss_utils.py`): Implements DPO/IPO/GRIPO losses reading from `config/loss/`.

e. Evaluation & Logging
- Eval modules (`src/eval/*.py`): Win‐rate and fast OAI evaluations for post‐training validation.
- WandB service (external): Receives training and evaluation metrics.

f. Orchestration Scripts
- `train.py`: Main CLI entrypoint. Parses args, loads config, calls trainer factory, kicks off training.
- Shell scripts (`scripts/*.sh`): Preconfigured invocations of `train.py` for SFT (`run_sft.sh`), IPO (`run_multi.sh`), GRIPO (`run_multi_robust.sh`).

g. Plotting Layer
- `plot_scripts/`: Python scripts that fetch WandB metrics via API and generate both full and paper‐style plots.

##### Relationships & Data Flows
- Config Loader → Data Layer & Model Layer & Training Layer (decisions flow from config to each component).
- Data Layer → Training Layer (feeds training data batches).
- Model Layer + Loss Utilities → Training Layer (model + loss define optimization).
- Training Layer → WandB (logs metrics) & Checkpoint Storage (local/remote).
- Training Layer → Evaluation Modules (optionally invoked mid‐training or post‐training).
- Plotting Layer ← WandB API (reads logged metrics).

##### Architectural Patterns & Principles
- Factory Pattern in `trainers_factory.py` for extensible trainer selection.
- Config‐Driven Design: Almost every behavior is driven by YAML files.
- Separation of Concerns: Data ingestion, modeling, training, evaluation, and plotting live in distinct directories.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Layer
    subgraph "Configuration Layer"
        direction TB
        ConfigDir["config/"]:::config
        ConfigLoader["Config Loader\n(src/utils.py)"]:::config
    end

    %% Orchestration Scripts
    subgraph "Orchestration"
        direction TB
        TrainCLI["train.py"]:::config
        run_sft["run_sft.sh"]:::config
        run_multi["run_multi.sh"]:::config
        run_multi_robust["run_multi_robust.sh"]:::config
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DataSelection["Data Selection\n(src/data_selection.py)"]:::data
        PrefDatasets["Preference Datasets\n(src/preference_datasets.py)"]:::data
        GroupProcessing["Group Data Processing\n(src/groupstuff/)"]:::data
    end

    %% Model & Loss Layer
    subgraph "Model & Loss Layer"
        direction TB
        Models["Model Definitions\n(src/models.py)"]:::model
        LossUtils["Loss Utilities\n(src/loss_utils.py)"]:::model
    end

    %% Training Layer
    subgraph "Training Layer"
        direction TB
        TrainerFactory["Trainer Factory\n(src/trainers_factory.py)"]:::training
        BasicTrainer["BasicTrainer\n(src/trainers/basictrainer.py)"]:::training
        ParallelTrainer["ParallelTrainer\n(src/trainers/paralleltrainer.py)"]:::training
        GroupTrainer["GroupTrainer\n(src/trainers/grouptrainer.py)"]:::training
        GroupEarlyStopTrainer["GroupEarlyStopTrainer\n(src/trainers/grouptrainerearlystop.py)"]:::training
    end

    %% Evaluation & Logging
    subgraph "Evaluation & Logging"
        direction TB
        FastOAI["Fast OAI Eval\n(src/eval/fast_oai.py)"]:::eval
        WinRate["Win-Rate Eval\n(src/eval/win_rate.py)"]:::eval
    end

    %% Plotting Layer
    subgraph "Plotting Layer"
        direction TB
        FullPlots["Full Metrics Plots\n(plot_scripts/plot_from_wandb_full_metrics.py)"]:::plotting
        PaperPlots["Paper-Style Plots\n(plot_scripts/plot_from_wandb_paper_plots.py)"]:::plotting
        WandBUtils["WandB API Utils\n(plot_scripts/visualisations_utils_wandb_api.py)"]:::plotting
    end

    %% External Services
    subgraph "External Services"
        direction TB
        HFHub["HuggingFace Model Hub"]:::external
        HFDatasets["HuggingFace Datasets"]:::external
        WandB["Weights & Biases"]:::external
    end

    %% Relationships
    ConfigDir -->|loads| ConfigLoader
    ConfigLoader -.->|provides config| DataSelection
    ConfigLoader -.->|provides config| Models
    ConfigLoader -.->|provides config| TrainerFactory

    TrainCLI -->|calls| ConfigLoader
    TrainCLI -->|invokes| TrainerFactory
    run_sft -->|executes| TrainCLI
    run_multi -->|executes| TrainCLI
    run_multi_robust -->|executes| TrainCLI

    DataSelection -->|batch data| TrainerFactory
    PrefDatasets -->|datasets| TrainerFactory
    GroupProcessing -->|grouped data| TrainerFactory

    Models -->|model instance| TrainerFactory
    LossUtils -->|loss funcs| TrainerFactory

    TrainerFactory -->|instantiates| BasicTrainer
    TrainerFactory -->|instantiates| ParallelTrainer
    TrainerFactory -->|instantiates| GroupTrainer
    TrainerFactory -->|instantiates| GroupEarlyStopTrainer

    BasicTrainer -->|logs metrics| WandB
    ParallelTrainer -->|logs metrics| WandB
    GroupTrainer -->|logs metrics| WandB
    GroupEarlyStopTrainer -->|logs metrics| WandB

    BasicTrainer -->|checkpoint & output| HFHub
    ParallelTrainer -->|checkpoint & output| HFHub
    GroupTrainer -->|checkpoint & output| HFHub
    GroupEarlyStopTrainer -->|checkpoint & output| HFHub

    BasicTrainer -->|trigger| FastOAI
    BasicTrainer -->|trigger| WinRate
    ParallelTrainer -->|trigger| FastOAI
    ParallelTrainer -->|trigger| WinRate
    GroupTrainer -->|trigger| FastOAI
    GroupTrainer -->|trigger| WinRate
    GroupEarlyStopTrainer -->|trigger| FastOAI
    GroupEarlyStopTrainer -->|trigger| WinRate

    WandB -->|metrics API| FullPlots
    WandB -->|metrics API| PaperPlots
    WandBUtils -->|uses| WandB

    %% Click Events
    click ConfigDir "https://github.com/rsshyam/grpo/tree/main/config/"
    click ConfigLoader "https://github.com/rsshyam/grpo/blob/main/src/utils.py"
    click TrainCLI "https://github.com/rsshyam/grpo/blob/main/train.py"
    click run_sft "https://github.com/rsshyam/grpo/blob/main/scripts/run_sft.sh"
    click run_multi "https://github.com/rsshyam/grpo/blob/main/scripts/run_multi.sh"
    click run_multi_robust "https://github.com/rsshyam/grpo/blob/main/scripts/run_multi_robust.sh"
    click DataSelection "https://github.com/rsshyam/grpo/blob/main/src/data_selection.py"
    click PrefDatasets "https://github.com/rsshyam/grpo/blob/main/src/preference_datasets.py"
    click GroupProcessing "https://github.com/rsshyam/grpo/tree/main/src/groupstuff/"
    click Models "https://github.com/rsshyam/grpo/blob/main/src/models.py"
    click TrainerFactory "https://github.com/rsshyam/grpo/blob/main/src/trainers_factory.py"
    click BasicTrainer "https://github.com/rsshyam/grpo/blob/main/src/trainers/basictrainer.py"
    click ParallelTrainer "https://github.com/rsshyam/grpo/blob/main/src/trainers/paralleltrainer.py"
    click GroupTrainer "https://github.com/rsshyam/grpo/blob/main/src/trainers/grouptrainer.py"
    click GroupEarlyStopTrainer "https://github.com/rsshyam/grpo/blob/main/src/trainers/grouptrainerearlystop.py"
    click LossUtils "https://github.com/rsshyam/grpo/blob/main/src/loss_utils.py"
    click FastOAI "https://github.com/rsshyam/grpo/blob/main/src/eval/fast_oai.py"
    click WinRate "https://github.com/rsshyam/grpo/blob/main/src/eval/win_rate.py"
    click FullPlots "https://github.com/rsshyam/grpo/blob/main/plot_scripts/plot_from_wandb_full_metrics.py"
    click PaperPlots "https://github.com/rsshyam/grpo/blob/main/plot_scripts/plot_from_wandb_paper_plots.py"
    click WandBUtils "https://github.com/rsshyam/grpo/blob/main/plot_scripts/visualisations_utils_wandb_api.py"

    %% Styles
    classDef config fill:#D6EAF8,stroke:#1F618D,color:#1F618D
    classDef data fill:#D5F5E3,stroke:#239B56,color:#239B56
    classDef model fill:#FDEBD0,stroke:#D35400,color:#D35400
    classDef training fill:#E8DAEF,stroke:#8E44AD,color:#8E44AD
    classDef eval fill:#D1F2EB,stroke:#117A65,color:#117A65
    classDef plotting fill:#F9E79F,stroke:#B7950B,color:#B7950B
    classDef external fill:#E5E7E9,stroke:#566573,color:#566573
```
