# phivandat123/preference-distillation architecture

> Architecture diagram of the GitHub repository phivandat123/preference-distillation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/phivandat123/preference-distillation
- Repository: https://github.com/phivandat123/preference-distillation

## Overview

Preference Distillation is a research codebase for preparing preference and response data, training teacher and student models, and applying several student-distillation objectives. The documented workflow starts with teacher/student SFT and teacher DPO, then prepares method-specific data and trains DCKD, TVKD, ADPA/ADPA+, CTPD, PPD, or VPD. PPD and VPD share generated multi-response data; DCKD/TVKD and ADPA use precomputed data. The README documents these stages and artifacts; sampled source confirms selected data, generation, and logit-processing behavior, but not every training implementation.

Main entry points: [`utils/precompute_logits.py`](https://github.com/phivandat123/preference-distillation/blob/main/utils/precompute_logits.py) (Logit precompute), [`run/run_dpo_teacher.sh`](https://github.com/phivandat123/preference-distillation/blob/main/run/run_dpo_teacher.sh) (Teacher DPO), [`data_gen/gen/generate_dataset.py`](https://github.com/phivandat123/preference-distillation/blob/main/data_gen/gen/generate_dataset.py) (PPD/VPD generation), [`run/run_sft_teacher.sh`](https://github.com/phivandat123/preference-distillation/blob/main/run/run_sft_teacher.sh) (Teacher SFT)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_precompute["Logit precompute"]
  node_train_logits["Training logits"]
  node_ctpd_prepare["CTPD preparation"]
  node_ppd_vpd_generate["PPD/VPD generation"]
  node_multi_response["Multi-response data"]
  node_dckd_data["DCKD/TVKD data"]
  node_adpa_data["ADPA data"]
  node_ctpd_data["CTPD data"]
end

subgraph group_base["Base model training"]
  node_teacher_sft["Teacher SFT<br/>[run_sft_teacher.sh]"]
  node_teacher_model["Teacher model"]
  node_teacher_dpo["Teacher DPO<br/>[run_dpo_teacher.sh]"]
  node_student_sft["Student SFT<br/>[run_sft_student.sh]"]
  node_student_model["Student model"]
end

subgraph group_distill["Distillation methods"]
  node_dckd["DCKD training<br/>[run_dckd.sh]"]
  node_tvkd["TVKD training<br/>[run_tvkd.sh]"]
  node_adpa["ADPA training<br/>[run_adpa.sh]"]
  node_adpa_plus["ADPA+ training<br/>[run_adpa_plus.sh]"]
  node_ctpd["CTPD training<br/>[run_ctpd.sh]"]
  node_ppd["PPD training<br/>[run_ppd.sh]"]
  node_vpd["VPD training<br/>[run_vpd.sh]"]
end

subgraph group_runtime["Shared runtime"]
  node_alignment_data["Dataset formatting<br/>[data.py]"]
  node_generation["Model generation<br/>[vllm_generate.py]"]
  node_compress["Logit compression<br/>[compress_logits.py]"]
end

node_researcher(("Researcher"))
node_hf_datasets[("HF datasets")]
node_outputs["Trained models"]

node_researcher -->|"launches"| node_teacher_sft
node_researcher -->|"launches"| node_teacher_dpo
node_researcher -->|"launches"| node_student_sft
node_researcher -->|"launches"| node_precompute
node_researcher -->|"launches"| node_ctpd_prepare
node_researcher -->|"launches"| node_ppd_vpd_generate
node_hf_datasets -.->|"provides data"| node_teacher_sft
node_hf_datasets -.->|"provides data"| node_teacher_dpo
node_hf_datasets -.->|"provides data"| node_student_sft
node_hf_datasets -.->|"provides data"| node_precompute
node_hf_datasets -.->|"provides data"| node_ppd_vpd_generate
node_teacher_sft -->|"writes model"| node_teacher_model
node_teacher_model -->|"initializes"| node_teacher_dpo
node_teacher_dpo -->|"writes model"| node_teacher_model
node_student_sft -->|"writes model"| node_student_model
node_teacher_model -->|"supplies model"| node_precompute
node_student_model -->|"supplies generator"| node_precompute
node_precompute -->|"writes data"| node_dckd_data
node_precompute -->|"writes data"| node_adpa_data
node_student_model -->|"supplies model"| node_train_logits
node_train_logits -->|"writes logits"| node_dckd_data
node_ctpd_prepare -->|"writes data"| node_ctpd_data
node_ppd_vpd_generate -->|"writes data"| node_multi_response
node_generation -->|"generates responses"| node_multi_response
node_alignment_data -->|"formats dataset"| node_dckd_data
node_dckd_data -->|"trains on"| node_dckd
node_dckd -->|"writes model"| node_student_model
node_dckd_data -->|"trains on"| node_tvkd
node_tvkd -->|"writes model"| node_outputs
node_adpa_data -->|"trains on"| node_adpa
node_adpa -->|"writes model"| node_outputs
node_adpa_data -->|"trains on"| node_adpa_plus
node_adpa_plus -->|"writes model"| node_outputs
node_ctpd_data -->|"trains on"| node_ctpd
node_ctpd -->|"writes model"| node_outputs
node_multi_response -->|"trains on"| node_ppd
node_ppd -->|"writes model"| node_outputs
node_multi_response -->|"trains on"| node_vpd
node_vpd -->|"writes model"| node_outputs
node_compress -->|"compresses logits"| node_dckd_data

click node_teacher_sft "https://github.com/phivandat123/preference-distillation/blob/main/run/run_sft_teacher.sh"
click node_teacher_dpo "https://github.com/phivandat123/preference-distillation/blob/main/run/run_dpo_teacher.sh"
click node_student_sft "https://github.com/phivandat123/preference-distillation/blob/main/run/run_sft_student.sh"
click node_precompute "https://github.com/phivandat123/preference-distillation/blob/main/utils/precompute_logits.py"
click node_train_logits "https://github.com/phivandat123/preference-distillation/blob/main/utils/precompute_logits_for_train.py"
click node_ctpd_prepare "https://github.com/phivandat123/preference-distillation/blob/main/utils/prepare_ctpd_dataset.py"
click node_ppd_vpd_generate "https://github.com/phivandat123/preference-distillation/blob/main/data_gen/gen/generate_dataset.py"
click node_dckd "https://github.com/phivandat123/preference-distillation/blob/main/run/run_dckd.sh"
click node_tvkd "https://github.com/phivandat123/preference-distillation/blob/main/run/run_tvkd.sh"
click node_adpa "https://github.com/phivandat123/preference-distillation/blob/main/run/run_adpa.sh"
click node_adpa_plus "https://github.com/phivandat123/preference-distillation/blob/main/run/run_adpa_plus.sh"
click node_ctpd "https://github.com/phivandat123/preference-distillation/blob/main/run/run_ctpd.sh"
click node_ppd "https://github.com/phivandat123/preference-distillation/blob/main/run/run_ppd.sh"
click node_vpd "https://github.com/phivandat123/preference-distillation/blob/main/run/run_vpd.sh"
click node_alignment_data "https://github.com/phivandat123/preference-distillation/blob/main/alignment/data.py"
click node_generation "https://github.com/phivandat123/preference-distillation/blob/main/utils/vllm_generate.py"
click node_compress "https://github.com/phivandat123/preference-distillation/blob/main/utils/compress_logits.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_precompute,node_train_logits,node_ctpd_prepare,node_ppd_vpd_generate,node_multi_response,node_dckd_data,node_adpa_data,node_ctpd_data toneBlue
class node_teacher_sft,node_teacher_model,node_teacher_dpo,node_student_sft,node_student_model,node_hf_datasets toneAmber
class node_dckd,node_tvkd,node_adpa,node_adpa_plus,node_ctpd,node_ppd,node_vpd toneMint
class node_alignment_data,node_generation,node_compress toneRose
class node_researcher,node_outputs toneIndigo
```

## Components

### Data preparation

- [Logit precompute](https://github.com/phivandat123/preference-distillation/blob/main/utils/precompute_logits.py): `utils/precompute_logits.py`
- [Training logits](https://github.com/phivandat123/preference-distillation/blob/main/utils/precompute_logits_for_train.py): `utils/precompute_logits_for_train.py`
- [CTPD preparation](https://github.com/phivandat123/preference-distillation/blob/main/utils/prepare_ctpd_dataset.py): `utils/prepare_ctpd_dataset.py`
- [PPD/VPD generation](https://github.com/phivandat123/preference-distillation/blob/main/data_gen/gen/generate_dataset.py): `data_gen/gen/generate_dataset.py`
- **Multi-response data**
- **DCKD/TVKD data**
- **ADPA data**
- **CTPD data**

### Base model training

- [Teacher SFT](https://github.com/phivandat123/preference-distillation/blob/main/run/run_sft_teacher.sh): `run/run_sft_teacher.sh`
- **Teacher model**
- [Teacher DPO](https://github.com/phivandat123/preference-distillation/blob/main/run/run_dpo_teacher.sh): `run/run_dpo_teacher.sh`
- [Student SFT](https://github.com/phivandat123/preference-distillation/blob/main/run/run_sft_student.sh): `run/run_sft_student.sh`
- **Student model**

### Distillation methods

- [DCKD training](https://github.com/phivandat123/preference-distillation/blob/main/run/run_dckd.sh): `run/run_dckd.sh`
- [TVKD training](https://github.com/phivandat123/preference-distillation/blob/main/run/run_tvkd.sh): `run/run_tvkd.sh`
- [ADPA training](https://github.com/phivandat123/preference-distillation/blob/main/run/run_adpa.sh): `run/run_adpa.sh`
- [ADPA+ training](https://github.com/phivandat123/preference-distillation/blob/main/run/run_adpa_plus.sh): `run/run_adpa_plus.sh`
- [CTPD training](https://github.com/phivandat123/preference-distillation/blob/main/run/run_ctpd.sh): `run/run_ctpd.sh`
- [PPD training](https://github.com/phivandat123/preference-distillation/blob/main/run/run_ppd.sh): `run/run_ppd.sh`
- [VPD training](https://github.com/phivandat123/preference-distillation/blob/main/run/run_vpd.sh): `run/run_vpd.sh`

### Shared runtime

- [Dataset formatting](https://github.com/phivandat123/preference-distillation/blob/main/alignment/data.py): `alignment/data.py`
- [Model generation](https://github.com/phivandat123/preference-distillation/blob/main/utils/vllm_generate.py): `utils/vllm_generate.py`
- [Logit compression](https://github.com/phivandat123/preference-distillation/blob/main/utils/compress_logits.py): `utils/compress_logits.py`

### Other components

- **Researcher**
- **HF datasets**
- **Trained models**

## Connections

- Researcher → Teacher SFT: launches
- Researcher → Teacher DPO: launches
- Researcher → Student SFT: launches
- Researcher → Logit precompute: launches
- Researcher → CTPD preparation: launches
- Researcher → PPD/VPD generation: launches
- HF datasets → Teacher SFT: provides data
- HF datasets → Teacher DPO: provides data
- HF datasets → Student SFT: provides data
- HF datasets → Logit precompute: provides data
- HF datasets → PPD/VPD generation: provides data
- Teacher SFT → Teacher model: writes model
- Teacher model → Teacher DPO: initializes
- Teacher DPO → Teacher model: writes model
- Student SFT → Student model: writes model
- Teacher model → Logit precompute: supplies model
- Student model → Logit precompute: supplies generator
- Logit precompute → DCKD/TVKD data: writes data
- Logit precompute → ADPA data: writes data
- Student model → Training logits: supplies model
- Training logits → DCKD/TVKD data: writes logits
- CTPD preparation → CTPD data: writes data
- PPD/VPD generation → Multi-response data: writes data
- Model generation → Multi-response data: generates responses
- Dataset formatting → DCKD/TVKD data: formats dataset
- DCKD/TVKD data → DCKD training: trains on
- DCKD training → Student model: writes model
- DCKD/TVKD data → TVKD training: trains on
- TVKD training → Trained models: writes model
- ADPA data → ADPA training: trains on
- ADPA training → Trained models: writes model
- ADPA data → ADPA+ training: trains on
- ADPA+ training → Trained models: writes model
- CTPD data → CTPD training: trains on
- CTPD training → Trained models: writes model
- Multi-response data → PPD training: trains on
- PPD training → Trained models: writes model
- Multi-response data → VPD training: trains on
- VPD training → Trained models: writes model
- Logit compression → DCKD/TVKD data: compresses logits