How santoshadabala/distill-align-llm works
1. What this repo is
- This is a research pipeline for studying whether alignment fine-tuning hurts factuality in an LLM.
- The architecture is centered on a small number of offline jobs that generate model checkpoints and evaluation artifacts, plus a separate Streamlit dashboard that visualizes the results.
2. Primary flow: data → training → outputs → analysis
- Raw and curated datasets live in data/.
- Training scripts in scripts/ produce aligned checkpoints and response logs.
- Evaluation scripts then consume stored responses and checkpoints, writing summaries into outputs/.
- The dashboard and paper figures are built from those saved artifacts rather than recomputing everything live.
3. Core library boundary: src/distill_align/
- src/distill_align/config/ manages run configuration and model-specific settings.
- src/distill_align/data/processor.py handles dataset ingestion/transformation.
- src/distill_align/models/loader.py abstracts loading base and fine-tuned models.
- src/distill_align/training/ contains the training primitives for SFT, DPO, RLHF-style, and related methods.
- src/distill_align/serving/ and src/distill_align/monitoring/ suggest a lightweight inference/telemetry layer, but the main repo workload is research evaluation rather than production serving.
4. Training subsystem
- The main training entry points are scripts/run_sft.py, scripts/run_dpo.py, scripts/run_bsft.py, scripts/ipo.py, scripts/simpo.py, and scripts/run_full_pipeline.py.
- These scripts operate on the model/data abstractions in src/distill_align/training/ and configuration files under configs/.
- The README indicates the important training path is SFT → DPO on Llama-3.1-8B, with QLoRA-style fine-tuning on a single GPU.
- scripts/run_sft_scaling.py and the multiple YAML configs indicate controlled sweeps over dataset size and epochs.
5. Evaluation subsystem
- scripts/eval_factuality.py, scripts/eval_factuality_v2.py, scripts/eval_factuality_all.py, scripts/eval_truthfulqa.py, scripts/eval_semantic.py, scripts/eval_token_probs.py, and scripts/eval_token_probs.py cover different metrics and benchmark families.
- These scripts mostly work on saved model answers and produce CSV/JSON summaries under outputs/.
- This makes the evaluation layer reproducible and decoupled from the training layer.
6. Judge / re-scoring boundary
- A major architectural feature is that factuality scoring is not hard-coded; it is delegated to external judge models through API calls.
- scripts/rejudge.py re-scores stored model responses with a stronger judge.
- scripts/audit_references.py audits the gold answers themselves for factual errors.
- scripts/human_annotation.py and the stored outputs/... files show that the repo also supports human-anchor comparisons.
- This creates a clear boundary between local model generation and remote model-based evaluation.
7. Specialized benchmark generation
- scripts/generate_benchmark.py and scripts/generate_eval_prompts.py build the factuality benchmark prompts.
- scripts/build_trap_set.py and scripts/trap_eval.py create and evaluate the “no correct answer” trap set used to measure abstention vs fabrication.
- scripts/technical_instructions.py, data/technical_instructions.jsonl, and related JSONL files indicate the benchmark is domain-specific and templated rather than generic QA.
8. Calibration and diagnostic analysis
- scripts/calibration_ptrue.py computes the P(True) probe described in the README.
- scripts/ece.py, scripts/probability_mass_analysis.py, scripts/attention_shift_analysis.py, scripts/token_rank_analysis.py, and scripts/temperature_sweep.py are diagnostic tools for understanding confidence, token-level behavior, and calibration.
- These analyses are largely post hoc and consume stored answers/logits rather than modifying the training loop.
9. Artifact and results layout
- outputs/ is the main artifact store and effectively acts as the integration layer for the whole project.
- It contains model-evaluation CSVs, summaries, calibration outputs, cross-method comparisons, trap results, TruthfulQA results, and judge outputs.
- The many subfolders show repeated experiment families: baseline, SFT, DPO, scaling, calibration, rejudge, and sprint-style ablations.
- The logs in the repo root (*_log.txt) appear to be run transcripts or experiment notes.
10. Dashboard subsystem
- dashboard/app.py is a standalone Streamlit app that reads precomputed results from dashboard/*.json and the outputs/ directory.
- dashboard/requirements.txt isolates the UI dependencies from the main training/evaluation stack.
- The dashboard is therefore a presentation layer over static experiment artifacts, not a live backend service.
11. Documentation and reproducibility
- Makefile is the main orchestration surface for common workflows like install, audit, rejudge, and trap evaluation.
- docs/RESULTS.md, CHANGELOG.md, and the notebooks in notebooks/ provide experiment narration and exploratory analysis.
- pyproject.toml and tests/ show this is a packaged Python project with basic unit coverage around config and data processing.
12. External dependencies that matter architecturally
- Training and inference are built on PyTorch plus Hugging Face Transformers/TRL/PEFT and bitsandbytes.
- Judge/rejudge/audit steps depend on external LLM APIs, specifically OpenAI and Anthropic, which are part of the evaluation pipeline.
- Streamlit/Plotly are only for visualization; they do not participate in model training or scoring logic.
13. How to draw the architecture
- A good diagram would show four layers: data assets, training jobs, evaluation/analysis jobs, and dashboard presentation.
- The important arrows are: data/*.jsonl → training scripts → model checkpoints/responses → outputs/ → evaluation and judge scripts → dashboard/figures.
- A separate external-service box should capture OpenAI/Anthropic judges used for re-scoring and reference auditing.
Main entry points: eval_factuality.py Factual eval · run_sft.py SFT run · run_dpo.py DPO run · manager.py Run config
Components 20
Assets & Config
Datasets, benchmark prompts, and run configs that feed the research pipeline.
Core Library
Reusable Python package for config, data, model, training, serving, and tracking.
- Run config · Loads and manages run configuration and model-specific settings.
src/distill_align/config/manager.py - Data proc · Dataset ingestion and transformation logic.
src/distill_align/data/processor.py - Model loader · Abstracts loading base and fine-tuned models.
src/distill_align/models/loader.py - Training core · Reusable training primitives for supervised, preference, and RLHF-style methods.
src/distill_align/training - Inference layer · Lightweight serving and API engine used for model interaction and diagnostics.
src/distill_align/serving - Tracking · Run tracking and monitoring support for experiments.
src/distill_align/tracking
Training Jobs
Offline scripts that fine-tune models and generate checkpoints or response logs.
- SFT run · Main supervised fine-tuning entry point.
scripts/run_sft.py - DPO run · Preference optimization entry point.
scripts/run_dpo.py - BSFT run · Alternative fine-tuning path used in experiment families.
scripts/run_bsft.py - Full pipeline · End-to-end job that chains training and artifact generation.
scripts/run_full_pipeline.py
Evaluation & Analysis
Post-training scoring, calibration, diagnostics, and benchmark comparisons built from saved artifacts.
- Factual eval · Generates factuality metrics from stored model responses.
scripts/eval_factuality.py - Rejudge & audit · Re-scores responses and audits reference answers using external judges.
scripts/rejudge.py - Benchmarks · Builds factuality, trap, and eval prompts for benchmark runs.
scripts/generate_benchmark.py - Diagnostics · Calibration, token, attention, probability-mass, and temperature analyses.
scripts/calibration_ptrue.py - TruthfulQA · TruthfulQA evaluation and summary generation from saved outputs.
scripts/eval_truthfulqa.py
Presentation
Static dashboard and paper-oriented figure generation over precomputed results.
External Judges
Remote LLM APIs used for re-scoring and reference auditing.
- LLM APIs · External OpenAI and Anthropic models used for scoring and auditing.
Connections 28
- Data sets → Data proc: ingest · Datasets are transformed into training and evaluation-ready examples.
- Configs → Run config: configure · YAML configs define model and experiment settings.
- Run config → SFT run: drives · Training jobs load run settings from the config manager.
- Run config → DPO run: drives · Training jobs load run settings from the config manager.
- Run config → BSFT run: drives · Training jobs load run settings from the config manager.
- Data proc → SFT run: feeds · Processed datasets are consumed by supervised fine-tuning.
- Data proc → DPO run: feeds · Processed datasets are consumed by preference optimization.
- Model loader → SFT run: loads · Training jobs load base and adapted models through the loader boundary.
- Model loader → DPO run: loads · Training jobs load base and adapted models through the loader boundary.
- Training core → SFT run: implements · SFT script uses shared training primitives.
- Training core → DPO run: implements · DPO script uses shared training primitives.
- Training core → BSFT run: implements · BSFT script uses shared training primitives.
- SFT run → Full pipeline: stage · Full pipeline orchestrates the supervised fine-tuning stage.
- DPO run → Full pipeline: stage · Full pipeline orchestrates the preference-optimization stage.
- Full pipeline → Factual eval: outputs · Pipeline artifacts are passed into evaluation jobs.
- SFT run → Factual eval: produces · Fine-tuned checkpoints and responses are later evaluated.
- DPO run → Factual eval: produces · Fine-tuned checkpoints and responses are later evaluated.
- Benchmarks → Factual eval: inputs · Benchmark prompt generation supplies evaluation sets.
- Benchmarks → TruthfulQA: inputs · Generated prompts support benchmark-specific evaluation.
- Factual eval → LLM APIs: judges · Factuality scoring can be delegated to external judge models.
- Rejudge & audit → LLM APIs: calls · Rejudging and reference audits use external API-backed models.
- Rejudge & audit → Factual eval: rescoring · Stored responses are rescored to refresh summary metrics.
- Diagnostics → Factual eval: analyzes · Calibration and token-level diagnostics consume saved evaluation artifacts.
- TruthfulQA → Dashboard: publishes · Benchmark summaries are surfaced in the dashboard.
- Factual eval → Dashboard: publishes · Evaluation outputs feed the presentation layer.
- Figures → Dashboard: shares artifacts · Figures are built from the same precomputed results used by the dashboard.
- Inference layer → Diagnostics: supports · Serving and inference utilities help generate logits and response traces.
- Tracking → Full pipeline: records · Experiment tracking captures job metadata and outputs.