# eric-mitchell/direct-preference-optimization architecture

> Architecture diagram of the GitHub repository eric-mitchell/direct-preference-optimization, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eric-mitchell/direct-preference-optimization
- Repository: https://github.com/eric-mitchell/direct-preference-optimization

## Overview

##### Project Type and Purpose
- This repository is a reference implementation for a two-stage language model training pipeline using Direct Preference Optimization (DPO). It supports both supervised fine-tuning (SFT) and DPO (or its variants like conservative DPO and IPO) for preference-based training.
- The system is designed to work with any causal HuggingFace model and is extendable by adding new datasets or model configuration files. Its main focus is on efficiently processing preference data for training and enabling distributed GPU training through various trainer classes.

##### Analyzing the File Structure
- Top-Level Files and Directories:
- The root directory includes configuration files (LICENSE, README, requirements.txt), main training scripts (train.py, trainers.py, utils.py, preference_datasets.py), and a "config" directory.
- The config directory:
- Contains a main config.yaml along with separate directories for “loss” and “model”. Each holds several YAML configuration files, describing various training settings (loss functions for SFT and DPO, different model configurations such as GPT2 variants, GPT-J, Llama, etc.).
- The training scripts:
- train.py: Acts as the main entry point handling both SFT and DPO training.
- trainers.py: Implements different trainer classes (BasicTrainer, FSDPTrainer, TensorParallelTrainer) to support various training strategies and multi-GPU distributions.
- preference_datasets.py: Contains dataset processing logic and is a key point for extension when adding custom datasets.
- utils.py: Provides shared helper functions used throughout the training pipeline.

##### Key Components
a. Main Components:
- Command Line Interface Entry (train.py): The central orchestration point that accepts runtime parameters (e.g., model type, dataset list, loss type, trainer choice, etc.).
- Configuration Layer: The YAML config files under config (including model and loss configurations). This layer dictates training hyperparameters, dataset configurations, and model specifications.
- Data Processing Module: Preference_datasets.py, which processes incoming dataset formats and maps prompts to responses and preference pairs.
- Trainer Classes Module: Found in trainers.py. This includes:
- BasicTrainer for single-GPU or naïve multi-GPU setups,
- FSDPTrainer for efficient multi-GPU training using Fully Sharded Data Parallel,
- TensorParallelTrainer for experimental parallelism.
- Util Functions: The utils.py file which supports various common operations.
- Logging/Monitoring: While not explicitly a module, references to wandb output logs (as mentioned in the README) should be noted, especially during training evaluation steps.

b. Relationships and Interactions:
- The CLI (train.py) uses configuration definitions from the config folder to set up training. It then passes parameters (including loss function parameters, model archive paths, etc.) to the chosen Trainer class.
- The Trainer classes from trainers.py encapsulate the training loop (SFT or DPO) while integrating components like mixed precision (via FSDP specifically) and activation checkpointing based on provided config options.
- During training, the dataset processing logic in preference_datasets.py prepares and feeds data (prompts, responses, preference pairs) into the appropriate training phase.
- The loss configurations (for both SFT and DPO) are defined in config/loss, and these dictate model optimization behavior which is used during training.

c. Architectural Patterns and Design Principles:
- Two-Stage Pipeline: Clearly showing a Stage 1 (SFT) and Stage 2 (DPO) with the output of Stage 1 (the trained model checkpoint) feeding into Stage 2.
- Modular and Configurable Design: The use of external YAML configuration files to determine model settings, loss parameters, and trainer specifics; this allows the system to be extended and reconfigured without changing main code logic.
- Distributed Training Support: Through modular trainer classes, particularly emphasizing multi-GPU support with FSDP and experimental tensor parallelism.

d. Relevant Technologies, Frameworks, and Libraries:
- HuggingFace Transformers (implied by reference to causal models and tokenizers).
- PyTorch (with specific support for Fully Sharded Data Parallel (FSDP) and mixed precision training).
- WandB for experiment logging and visualization.
- YAML processing for configuration management.

##### Specific Elements
- For example, "Configuration Layer" for config files, "Main Orchestration" for train.py.
- From command-line input → train.py → Trainer instantiation → Data Loader (preference_datasets.py) → Training Loop (inside trainers) → Checkpoint saving.
- External dependencies:
- Model definitions (HuggingFace models) and experimental logging (wandb).
- Optionally, add a note on how mixed precision and activation checkpointing come into play (especially with FSDPTrainer).

##### High-Level Suggestions
- That every component (especially the configuration files and dataset handling functions) is represented so that the overall flow from data input to training output is clear.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI and Setup
    CLI["CLI Interface - train.py"]:::cli
    Utils["Utility Functions - utils.py"]:::utils

    %% Configuration Layer
    MainConfig["Main Configuration - config/config.yaml"]:::config
    LossConfig["Loss Configurations - config/loss"]:::config
    ModelConfig["Model Configurations - config/model"]:::config

    %% Data and Trainers
    Data["Data Handler - preference_datasets.py"]:::data
    Trainer["Trainer Classes - trainers.py"]:::trainer
    subgraph "Trainer Variants"
        BT["BasicTrainer"]:::trainerVariant
        FSDP["FSDPTrainer"]:::trainerVariant
        TPT["TensorParallelTrainer"]:::trainerVariant
    end

    %% Training Stages
    SFT["Stage 1: Supervised Fine-Tuning (SFT)"]:::stage
    Checkpoint["Checkpoint: Trained Model"]:::checkpoint
    DPO["Stage 2: Direct Preference Optimization (DPO)"]:::stage

    %% External Dependencies
    subgraph "External Dependencies"
        HF["HuggingFace Transformers"]:::external
        PT["PyTorch (FSDP,MixedPrecision)"]:::external
        WAN["WandB Logging"]:::external
    end

    %% Connections - Flow of control and data
    CLI -->|"loads"| MainConfig
    CLI -->|"loads"| LossConfig
    CLI -->|"loads"| ModelConfig

    MainConfig -->|"configures"| Trainer
    LossConfig -->|"configures"| Trainer
    ModelConfig -->|"configures"| Trainer

    Data -->|"feedsData"| Trainer

    Trainer -->|"runs"| SFT
    SFT -->|"produces"| Checkpoint
    Checkpoint -->|"feeds"| DPO

    %% Trainer variant relationships (internal to Trainer Classes)
    Trainer --> BT
    Trainer --> FSDP
    Trainer --> TPT

    %% External dependencies usage
    Trainer -->|"uses"| HF
    Trainer -->|"uses"| PT
    Trainer -->|"logsTo"| WAN

    %% Utility Functions supporting CLI and Trainer operations
    CLI --- Utils
    Trainer --- Utils

    %% Styling and Colors
    classDef cli fill:#fef3c7,stroke:#d97706,stroke-width:2px;
    classDef utils fill:#e0f2fe,stroke:#0284c7,stroke-width:2px;
    classDef config fill:#d1fae5,stroke:#047857,stroke-width:2px;
    classDef data fill:#e9d5ff,stroke:#9333ea,stroke-width:2px;
    classDef trainer fill:#e0e7ff,stroke:#4338ca,stroke-width:2px;
    classDef trainerVariant fill:#fce7f3,stroke:#db2777,stroke-width:2px;
    classDef stage fill:#fee2e2,stroke:#b91c1c,stroke-width:2px;
    classDef checkpoint fill:#d1d5db,stroke:#6b7280,stroke-width:2px;
    classDef external fill:#cffafe,stroke:#0891b2,stroke-width:2px;

    %% Click Events for Component Mapping
    click CLI "https://github.com/eric-mitchell/direct-preference-optimization/blob/main/train.py"
    click MainConfig "https://github.com/eric-mitchell/direct-preference-optimization/blob/main/config/config.yaml"
    click LossConfig "https://github.com/eric-mitchell/direct-preference-optimization/tree/main/config/loss"
    click ModelConfig "https://github.com/eric-mitchell/direct-preference-optimization/tree/main/config/model"
    click Data "https://github.com/eric-mitchell/direct-preference-optimization/blob/main/preference_datasets.py"
    click Trainer "https://github.com/eric-mitchell/direct-preference-optimization/blob/main/trainers.py"
    click Utils "https://github.com/eric-mitchell/direct-preference-optimization/blob/main/utils.py"
```
