How eric-mitchell/direct-preference-optimization works
Project Type and Purpose
• This repository is a reference implementation for a two-stage language model training pipeline using Direct Preference Optimization (DPO). It supports both supervised fine-tuning (SFT) and DPO (or its variants like conservative DPO and IPO) for preference-based training.
• The system is designed to work with any causal HuggingFace model and is extendable by adding new datasets or model configuration files. Its main focus is on efficiently processing preference data for training and enabling distributed GPU training through various trainer classes.
Analyzing the File Structure
• Top-Level Files and Directories:
• The root directory includes configuration files (LICENSE, README, requirements.txt), main training scripts (train.py, trainers.py, utils.py, preference_datasets.py), and a "config" directory.
• The config directory:
• Contains a main config.yaml along with separate directories for “loss” and “model”. Each holds several YAML configuration files, describing various training settings (loss functions for SFT and DPO, different model configurations such as GPT2 variants, GPT-J, Llama, etc.).
• The training scripts:
• train.py: Acts as the main entry point handling both SFT and DPO training.
• trainers.py: Implements different trainer classes (BasicTrainer, FSDPTrainer, TensorParallelTrainer) to support various training strategies and multi-GPU distributions.
• preference_datasets.py: Contains dataset processing logic and is a key point for extension when adding custom datasets.
• utils.py: Provides shared helper functions used throughout the training pipeline.
Key Components
a. Main Components:
• Command Line Interface Entry (train.py): The central orchestration point that accepts runtime parameters (e.g., model type, dataset list, loss type, trainer choice, etc.).
• Configuration Layer: The YAML config files under config (including model and loss configurations). This layer dictates training hyperparameters, dataset configurations, and model specifications.
• Data Processing Module: Preference_datasets.py, which processes incoming dataset formats and maps prompts to responses and preference pairs.
• Trainer Classes Module: Found in trainers.py. This includes:
• BasicTrainer for single-GPU or naïve multi-GPU setups,
• FSDPTrainer for efficient multi-GPU training using Fully Sharded Data Parallel,
• TensorParallelTrainer for experimental parallelism.
• Util Functions: The utils.py file which supports various common operations.
• Logging/Monitoring: While not explicitly a module, references to wandb output logs (as mentioned in the README) should be noted, especially during training evaluation steps.
b. Relationships and Interactions:
• The CLI (train.py) uses configuration definitions from the config folder to set up training. It then passes parameters (including loss function parameters, model archive paths, etc.) to the chosen Trainer class.
• The Trainer classes from trainers.py encapsulate the training loop (SFT or DPO) while integrating components like mixed precision (via FSDP specifically) and activation checkpointing based on provided config options.
• During training, the dataset processing logic in preference_datasets.py prepares and feeds data (prompts, responses, preference pairs) into the appropriate training phase.
• The loss configurations (for both SFT and DPO) are defined in config/loss, and these dictate model optimization behavior which is used during training.
c. Architectural Patterns and Design Principles:
• Two-Stage Pipeline: Clearly showing a Stage 1 (SFT) and Stage 2 (DPO) with the output of Stage 1 (the trained model checkpoint) feeding into Stage 2.
• Modular and Configurable Design: The use of external YAML configuration files to determine model settings, loss parameters, and trainer specifics; this allows the system to be extended and reconfigured without changing main code logic.
• Distributed Training Support: Through modular trainer classes, particularly emphasizing multi-GPU support with FSDP and experimental tensor parallelism.
d. Relevant Technologies, Frameworks, and Libraries:
• HuggingFace Transformers (implied by reference to causal models and tokenizers).
• PyTorch (with specific support for Fully Sharded Data Parallel (FSDP) and mixed precision training).
• WandB for experiment logging and visualization.
• YAML processing for configuration management.
Specific Elements
• For example, "Configuration Layer" for config files, "Main Orchestration" for train.py.
• From command-line input → train.py → Trainer instantiation → Data Loader (preference_datasets.py) → Training Loop (inside trainers) → Checkpoint saving.
• External dependencies:
• Model definitions (HuggingFace models) and experimental logging (wandb).
• Optionally, add a note on how mixed precision and activation checkpointing come into play (especially with FSDPTrainer).
High-Level Suggestions
• That every component (especially the configuration files and dataset handling functions) is represented so that the overall flow from data input to training output is clear.