How chenyuxin1999/s-dpo works
Key Components
• Data Preparation & Ingestion:
• Files under sample_data (e.g., lastfm-train.json, lastfm-test.json, lastfm-val.json) show where the LastFM dataset is stored.
• Scripts like prepare_sft_data.py and lastfm_data.py indicate a pre-processing stage to prepare data for training.
• The prompt directory (with book.txt, movie.txt, music.txt) serves as templates or examples for converting past interactions to natural language prompts.
• Training Modules:
• There are three main training scripts represented by sft.py (Supervised Fine-Tuning), dpo.py (Direct Preference Optimization), and softmax_dpo.py (Softmax DPO).
• Corresponding shell scripts (sft.sh, dpo.sh, softmax_dpo.sh) function as entry points to launch each training regime.
• The trainer directory contains trainer-specific components (e.g., softmax_dpo_trainer.py and utils.py) – hinting at modular training utilities and logic encapsulated for S-DPO.
• Inference & Evaluation:
• inference.py and inference.sh deal with running the inference process and obtaining performance metrics.
• The evaluate_batch.py script likely helps in evaluating batches of recommendations.
• Underlying Infrastructure & Dependencies:
• The README mentions setup with PyTorch (Python 3.9.7 with PyTorch 2.2.2+cu117) which is a key dependency here.
• The requirements.txt file holds all external packages required, making it a backbone for dependency management.
• Assets and README images (like assets/framework.png) might be used to illustrate the overall architecture or framework in the documentation.
Relationships and Interactions
• The system follows a data-centric pipeline:
• The Data Preparation module loads and processes raw data from the sample_data folder and prompt templates.
• Once the data is prepared, one of the training modules (SFT, DPO, or S-DPO) is executed, depending on the experimental setup. Each training mode uses different algorithms but eventually aims to fine-tune a language model to make ranking predictions.
• The trained model is then subjected to Inference where its performance is evaluated using inference.py and evaluate_batch.py.
• Directional flows:
• Data flows -> Preprocessing (prepare_sft_data.py, lastfm_data.py) -> Training (via sft.py, dpo.py, or softmax_dpo.py) -> Inference/Evaluation.
• Trainer utilities (found under the trainer directory) assist the S-DPO training process and can be shown as sub-modules connected to the softmax_dpo component.
Architectural Patterns & Design Principles
• Pipeline Architecture: The project is structured as a sequential pipeline where input data is preprocessed, then fed into one of several training strategies, and finally evaluated.
• Modularity & Separation of Concerns: By having separate scripts and directories for prompt generation, data preparation, training, and inference, the design reflects a strong separation between different concerns.
• Script-based Workflow: The use of multiple shell scripts (.sh) orchestrates the running of different training and evaluation modes.
Relevant Technologies or Libraries
• PyTorch for model training.
• Custom Python modules that implement the different training objectives (SFT, DPO, S-DPO).
• Utility modules in the trainer folder assist in handling the training routine, loss calculations, and possibly other model-related utilities.