# chenyuxin1999/s-dpo architecture

> Architecture diagram of the GitHub repository chenyuxin1999/s-dpo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chenyuxin1999/s-dpo
- Repository: https://github.com/chenyuxin1999/s-dpo

## Overview

##### Key Components
- Data Preparation & Ingestion:
- Files under sample_data (e.g., lastfm-train.json, lastfm-test.json, lastfm-val.json) show where the LastFM dataset is stored.
- Scripts like prepare_sft_data.py and lastfm_data.py indicate a pre-processing stage to prepare data for training.
- The prompt directory (with book.txt, movie.txt, music.txt) serves as templates or examples for converting past interactions to natural language prompts.

- Training Modules:
- There are three main training scripts represented by sft.py (Supervised Fine-Tuning), dpo.py (Direct Preference Optimization), and softmax_dpo.py (Softmax DPO).
- Corresponding shell scripts (sft.sh, dpo.sh, softmax_dpo.sh) function as entry points to launch each training regime.
- The trainer directory contains trainer-specific components (e.g., softmax_dpo_trainer.py and utils.py) – hinting at modular training utilities and logic encapsulated for S-DPO.

- Inference & Evaluation:
- inference.py and inference.sh deal with running the inference process and obtaining performance metrics.
- The evaluate_batch.py script likely helps in evaluating batches of recommendations.

- Underlying Infrastructure & Dependencies:
- The README mentions setup with PyTorch (Python 3.9.7 with PyTorch 2.2.2+cu117) which is a key dependency here.
- The requirements.txt file holds all external packages required, making it a backbone for dependency management.
- Assets and README images (like assets/framework.png) might be used to illustrate the overall architecture or framework in the documentation.

##### Relationships and Interactions
- The system follows a data-centric pipeline:
- The Data Preparation module loads and processes raw data from the sample_data folder and prompt templates.
- Once the data is prepared, one of the training modules (SFT, DPO, or S-DPO) is executed, depending on the experimental setup. Each training mode uses different algorithms but eventually aims to fine-tune a language model to make ranking predictions.
- The trained model is then subjected to Inference where its performance is evaluated using inference.py and evaluate_batch.py.

- Directional flows:
- Data flows -> Preprocessing (prepare_sft_data.py, lastfm_data.py) -> Training (via sft.py, dpo.py, or softmax_dpo.py) -> Inference/Evaluation.
- Trainer utilities (found under the trainer directory) assist the S-DPO training process and can be shown as sub-modules connected to the softmax_dpo component.

##### Architectural Patterns & Design Principles
- Pipeline Architecture: The project is structured as a sequential pipeline where input data is preprocessed, then fed into one of several training strategies, and finally evaluated.
- Modularity & Separation of Concerns: By having separate scripts and directories for prompt generation, data preparation, training, and inference, the design reflects a strong separation between different concerns.
- Script-based Workflow: The use of multiple shell scripts (.sh) orchestrates the running of different training and evaluation modes.

##### Relevant Technologies or Libraries
- PyTorch for model training.
- Custom Python modules that implement the different training objectives (SFT, DPO, S-DPO).
- Utility modules in the trainer folder assist in handling the training routine, loss calculations, and possibly other model-related utilities.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preparation & Ingestion
    subgraph "Data Preparation & Ingestion"
        DS["Sample Data (lastfm-sft-cans20)"]:::data
        PT["Prompt Templates"]:::data
        PS1["prepare_sft_data.py"]:::data
        PS2["lastfm_data.py"]:::data
        PD["Processed Data"]:::data
        DS --> PS1
        PT --> PS2
        PS1 --> PD
        PS2 --> PD
    end

    %% Training Modules
    subgraph "Training Modules"
        SFT["Supervised Fine-Tuning (SFT)\n(sft.py, sft.sh)"]:::training
        DPO["Direct Preference Optimization (DPO)\n(dpo.py, dpo.sh)"]:::training
        SDPO["Softmax DPO (S-DPO)\n(softmax_dpo.py, softmax_dpo.sh)"]:::training
        TR["Trainer Utilities\n(softmax_dpo_trainer.py, utils.py)"]:::training
        TM["Trained Model"]:::training
        PD --> SFT
        PD --> DPO
        PD --> SDPO
        SDPO --> TR
        SFT --> TM
        DPO --> TM
        SDPO --> TM
    end

    %% Inference & Evaluation
    subgraph "Inference & Evaluation"
        IN["Inference Engine\n(inference.py, inference.sh)"]:::inference
        EV["Evaluation\n(evaluate_batch.py)"]:::inference
        TM --> IN
        IN --> EV
    end

    %% Infrastructure & Dependencies
    INF["Infrastructure & Dependencies\n(requirements.txt, README.md, assets)"]:::infra
    INF -.-> PD
    INF -.-> TM
    INF -.-> IN

    %% Click Events
    click DS "https://github.com/chenyuxin1999/s-dpo/tree/main/sample_data/lastfm-sft-cans20"
    click PT "https://github.com/chenyuxin1999/s-dpo/tree/main/prompt/"
    click PS1 "https://github.com/chenyuxin1999/s-dpo/blob/main/sample_data/prepare_sft_data.py"
    click PS2 "https://github.com/chenyuxin1999/s-dpo/blob/main/sample_data/lastfm_data.py"
    click SFT "https://github.com/chenyuxin1999/s-dpo/blob/main/sft.py"
    click DPO "https://github.com/chenyuxin1999/s-dpo/blob/main/dpo.py"
    click SDPO "https://github.com/chenyuxin1999/s-dpo/blob/main/softmax_dpo.py"
    click TR "https://github.com/chenyuxin1999/s-dpo/blob/main/trainer/softmax_dpo_trainer.py"
    click IN "https://github.com/chenyuxin1999/s-dpo/blob/main/inference.py"
    click EV "https://github.com/chenyuxin1999/s-dpo/blob/main/evaluate_batch.py"
    click INF "https://github.com/chenyuxin1999/s-dpo/blob/main/requirements.txt"

    %% Styles
    classDef data fill:#AED6F1,stroke:#3498DB,stroke-width:2px;
    classDef training fill:#F9E79F,stroke:#F39C12,stroke-width:2px;
    classDef inference fill:#A9DFBF,stroke:#27AE60,stroke-width:2px;
    classDef infra fill:#F5CBA7,stroke:#E67E22,stroke-width:2px;
```
