How phivandat123/preference-distillation works
Preference Distillation is a research codebase for preparing preference and response data, training teacher and student models, and applying several student-distillation objectives. The documented workflow starts with teacher/student SFT and teacher DPO, then prepares method-specific data and trains DCKD, TVKD, ADPA/ADPA+, CTPD, PPD, or VPD. PPD and VPD share generated multi-response data; DCKD/TVKD and ADPA use precomputed data. The README documents these stages and artifacts; sampled source confirms selected data, generation, and logit-processing behavior, but not every training implementation.
Main entry points: precompute_logits.py Logit precompute · run_dpo_teacher.sh Teacher DPO · generate_dataset.py PPD/VPD generation · run_sft_teacher.sh Teacher SFT
Components 26
Data preparation
- Logit precompute
utils/precompute_logits.py - Training logits
utils/precompute_logits_for_train.py - CTPD preparation
utils/prepare_ctpd_dataset.py - PPD/VPD generation
data_gen/gen/generate_dataset.py - Multi-response data
- DCKD/TVKD data
- ADPA data
- CTPD data
Base model training
- Teacher SFT
run/run_sft_teacher.sh - Teacher model
- Teacher DPO
run/run_dpo_teacher.sh - Student SFT
run/run_sft_student.sh - Student model
Distillation methods
- DCKD training
run/run_dckd.sh - TVKD training
run/run_tvkd.sh - ADPA training
run/run_adpa.sh - ADPA+ training
run/run_adpa_plus.sh - CTPD training
run/run_ctpd.sh - PPD training
run/run_ppd.sh - VPD training
run/run_vpd.sh
Shared runtime
- Dataset formatting
alignment/data.py - Model generation
utils/vllm_generate.py - Logit compression
utils/compress_logits.py
Other components
- Researcher
- HF datasets
- Trained models
Connections 40
- Researcher → Teacher SFT: launches
- Researcher → Teacher DPO: launches
- Researcher → Student SFT: launches
- Researcher → Logit precompute: launches
- Researcher → CTPD preparation: launches
- Researcher → PPD/VPD generation: launches
- HF datasets → Teacher SFT: provides data
- HF datasets → Teacher DPO: provides data
- HF datasets → Student SFT: provides data
- HF datasets → Logit precompute: provides data
- HF datasets → PPD/VPD generation: provides data
- Teacher SFT → Teacher model: writes model
- Teacher model → Teacher DPO: initializes
- Teacher DPO → Teacher model: writes model
- Student SFT → Student model: writes model
- Teacher model → Logit precompute: supplies model
- Student model → Logit precompute: supplies generator
- Logit precompute → DCKD/TVKD data: writes data
- Logit precompute → ADPA data: writes data
- Student model → Training logits: supplies model
- Training logits → DCKD/TVKD data: writes logits
- CTPD preparation → CTPD data: writes data
- PPD/VPD generation → Multi-response data: writes data
- Model generation → Multi-response data: generates responses
- Dataset formatting → DCKD/TVKD data: formats dataset
- DCKD/TVKD data → DCKD training: trains on
- DCKD training → Student model: writes model
- DCKD/TVKD data → TVKD training: trains on
- TVKD training → Trained models: writes model
- ADPA data → ADPA training: trains on
- ADPA training → Trained models: writes model
- ADPA data → ADPA+ training: trains on
- ADPA+ training → Trained models: writes model
- CTPD data → CTPD training: trains on
- CTPD training → Trained models: writes model
- Multi-response data → PPD training: trains on
- PPD training → Trained models: writes model
- Multi-response data → VPD training: trains on
- VPD training → Trained models: writes model
- Logit compression → DCKD/TVKD data: compresses logits