# santoshadabala/distill-align-llm architecture

> Architecture diagram of the GitHub repository santoshadabala/distill-align-llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/santoshadabala/distill-align-llm
- Repository: https://github.com/santoshadabala/distill-align-llm

## Overview

1. **What this repo is**
   - This is a research pipeline for studying whether alignment fine-tuning hurts factuality in an LLM.
   - The architecture is centered on a small number of offline jobs that generate model checkpoints and evaluation artifacts, plus a separate Streamlit dashboard that visualizes the results.

2. **Primary flow: data → training → outputs → analysis**
   - Raw and curated datasets live in `data/`.
   - Training scripts in `scripts/` produce aligned checkpoints and response logs.
   - Evaluation scripts then consume stored responses and checkpoints, writing summaries into `outputs/`.
   - The dashboard and paper figures are built from those saved artifacts rather than recomputing everything live.

3. **Core library boundary: `src/distill_align/`**
   - `src/distill_align/config/` manages run configuration and model-specific settings.
   - `src/distill_align/data/processor.py` handles dataset ingestion/transformation.
   - `src/distill_align/models/loader.py` abstracts loading base and fine-tuned models.
   - `src/distill_align/training/` contains the training primitives for SFT, DPO, RLHF-style, and related methods.
   - `src/distill_align/serving/` and `src/distill_align/monitoring/` suggest a lightweight inference/telemetry layer, but the main repo workload is research evaluation rather than production serving.

4. **Training subsystem**
   - The main training entry points are `scripts/run_sft.py`, `scripts/run_dpo.py`, `scripts/run_bsft.py`, `scripts/ipo.py`, `scripts/simpo.py`, and `scripts/run_full_pipeline.py`.
   - These scripts operate on the model/data abstractions in `src/distill_align/training/` and configuration files under `configs/`.
   - The README indicates the important training path is SFT → DPO on Llama-3.1-8B, with QLoRA-style fine-tuning on a single GPU.
   - `scripts/run_sft_scaling.py` and the multiple YAML configs indicate controlled sweeps over dataset size and epochs.

5. **Evaluation subsystem**
   - `scripts/eval_factuality.py`, `scripts/eval_factuality_v2.py`, `scripts/eval_factuality_all.py`, `scripts/eval_truthfulqa.py`, `scripts/eval_semantic.py`, `scripts/eval_token_probs.py`, and `scripts/eval_token_probs.py` cover different metrics and benchmark families.
   - These scripts mostly work on saved model answers and produce CSV/JSON summaries under `outputs/`.
   - This makes the evaluation layer reproducible and decoupled from the training layer.

6. **Judge / re-scoring boundary**
   - A major architectural feature is that factuality scoring is not hard-coded; it is delegated to external judge models through API calls.
   - `scripts/rejudge.py` re-scores stored model responses with a stronger judge.
   - `scripts/audit_references.py` audits the gold answers themselves for factual errors.
   - `scripts/human_annotation.py` and the stored `outputs/...` files show that the repo also supports human-anchor comparisons.
   - This creates a clear boundary between local model generation and remote model-based evaluation.

7. **Specialized benchmark generation**
   - `scripts/generate_benchmark.py` and `scripts/generate_eval_prompts.py` build the factuality benchmark prompts.
   - `scripts/build_trap_set.py` and `scripts/trap_eval.py` create and evaluate the “no correct answer” trap set used to measure abstention vs fabrication.
   - `scripts/technical_instructions.py`, `data/technical_instructions.jsonl`, and related JSONL files indicate the benchmark is domain-specific and templated rather than generic QA.

8. **Calibration and diagnostic analysis**
   - `scripts/calibration_ptrue.py` computes the P(True) probe described in the README.
   - `scripts/ece.py`, `scripts/probability_mass_analysis.py`, `scripts/attention_shift_analysis.py`, `scripts/token_rank_analysis.py`, and `scripts/temperature_sweep.py` are diagnostic tools for understanding confidence, token-level behavior, and calibration.
   - These analyses are largely post hoc and consume stored answers/logits rather than modifying the training loop.

9. **Artifact and results layout**
   - `outputs/` is the main artifact store and effectively acts as the integration layer for the whole project.
   - It contains model-evaluation CSVs, summaries, calibration outputs, cross-method comparisons, trap results, TruthfulQA results, and judge outputs.
   - The many subfolders show repeated experiment families: baseline, SFT, DPO, scaling, calibration, rejudge, and sprint-style ablations.
   - The logs in the repo root (`*_log.txt`) appear to be run transcripts or experiment notes.

10. **Dashboard subsystem**
   - `dashboard/app.py` is a standalone Streamlit app that reads precomputed results from `dashboard/*.json` and the `outputs/` directory.
   - `dashboard/requirements.txt` isolates the UI dependencies from the main training/evaluation stack.
   - The dashboard is therefore a presentation layer over static experiment artifacts, not a live backend service.

11. **Documentation and reproducibility**
   - `Makefile` is the main orchestration surface for common workflows like install, audit, rejudge, and trap evaluation.
   - `docs/RESULTS.md`, `CHANGELOG.md`, and the notebooks in `notebooks/` provide experiment narration and exploratory analysis.
   - `pyproject.toml` and `tests/` show this is a packaged Python project with basic unit coverage around config and data processing.

12. **External dependencies that matter architecturally**
   - Training and inference are built on PyTorch plus Hugging Face Transformers/TRL/PEFT and bitsandbytes.
   - Judge/rejudge/audit steps depend on external LLM APIs, specifically OpenAI and Anthropic, which are part of the evaluation pipeline.
   - Streamlit/Plotly are only for visualization; they do not participate in model training or scoring logic.

13. **How to draw the architecture**
   - A good diagram would show four layers: data assets, training jobs, evaluation/analysis jobs, and dashboard presentation.
   - The important arrows are: `data/*.jsonl` → training scripts → model checkpoints/responses → `outputs/` → evaluation and judge scripts → dashboard/figures.
   - A separate external-service box should capture OpenAI/Anthropic judges used for re-scoring and reference auditing.

Main entry points: [`scripts/eval_factuality.py`](https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/eval_factuality.py) (Factual eval), [`scripts/run_sft.py`](https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/run_sft.py) (SFT run), [`scripts/run_dpo.py`](https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/run_dpo.py) (DPO run), [`src/distill_align/config/manager.py`](https://github.com/santoshadabala/distill-align-llm/blob/main/src/distill_align/config/manager.py) (Run config)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["Assets & Config"]
  node_data_assets[("Data sets<br/>jsonl corpora")]
  node_configs["Configs<br/>yaml runs"]
end

subgraph group_g2["Core Library"]
  node_src_config["Run config<br/>config manager<br/>[manager.py]"]
  node_src_data["Data proc<br/>dataset processing<br/>[processor.py]"]
  node_src_models["Model loader<br/>checkpoint loader<br/>[loader.py]"]
  node_src_train["Training core<br/>sft/dpo/rlhf"]
  node_src_serving["Inference layer<br/>serving/api"]
  node_src_tracking["Tracking<br/>telemetry"]
end

subgraph group_g3["Training Jobs"]
  node_run_sft["SFT run<br/>training script<br/>[run_sft.py]"]
  node_run_dpo["DPO run<br/>training script<br/>[run_dpo.py]"]
  node_run_bsft["BSFT run<br/>training script<br/>[run_bsft.py]"]
  node_run_full["Full pipeline<br/>orchestrator"]
end

subgraph group_g4["Evaluation & Analysis"]
  node_eval_factuality["Factual eval<br/>scored evaluation<br/>[eval_factuality.py]"]
  node_judge_tools["Rejudge & audit<br/>judge pipeline<br/>[rejudge.py]"]
  node_benchmarks["Benchmarks<br/>prompt generation"]
  node_diagnostics["Diagnostics<br/>analysis suite"]
  node_truthfulqa["TruthfulQA<br/>benchmark eval<br/>[eval_truthfulqa.py]"]
end

subgraph group_g5["Presentation"]
  node_dashboard_app["Dashboard<br/>streamlit app<br/>[app.py]"]
  node_figure_maker["Figures<br/>reporting<br/>[make_figures.py]"]
end

subgraph group_g6["External Judges"]
  node_openai_anthropic[("LLM APIs<br/>remote judges")]
end

node_data_assets -->|"ingest"| node_src_data
node_configs -->|"configure"| node_src_config
node_src_config -->|"drives"| node_run_sft
node_src_config -->|"drives"| node_run_dpo
node_src_config -->|"drives"| node_run_bsft
node_src_data -->|"feeds"| node_run_sft
node_src_data -->|"feeds"| node_run_dpo
node_src_models -->|"loads"| node_run_sft
node_src_models -->|"loads"| node_run_dpo
node_src_train -->|"implements"| node_run_sft
node_src_train -->|"implements"| node_run_dpo
node_src_train -->|"implements"| node_run_bsft
node_run_sft -->|"stage"| node_run_full
node_run_dpo -->|"stage"| node_run_full
node_run_full -->|"outputs"| node_eval_factuality
node_run_sft -->|"produces"| node_eval_factuality
node_run_dpo -->|"produces"| node_eval_factuality
node_benchmarks -->|"inputs"| node_eval_factuality
node_benchmarks -->|"inputs"| node_truthfulqa
node_eval_factuality -->|"judges"| node_openai_anthropic
node_judge_tools -->|"calls"| node_openai_anthropic
node_judge_tools -->|"rescoring"| node_eval_factuality
node_diagnostics -->|"analyzes"| node_eval_factuality
node_truthfulqa -->|"publishes"| node_dashboard_app
node_eval_factuality -->|"publishes"| node_dashboard_app
node_figure_maker -->|"shares artifacts"| node_dashboard_app
node_src_serving -->|"supports"| node_diagnostics
node_src_tracking -->|"records"| node_run_full

click node_data_assets "https://github.com/santoshadabala/distill-align-llm/tree/main/data"
click node_configs "https://github.com/santoshadabala/distill-align-llm/tree/main/configs"
click node_src_config "https://github.com/santoshadabala/distill-align-llm/blob/main/src/distill_align/config/manager.py"
click node_src_data "https://github.com/santoshadabala/distill-align-llm/blob/main/src/distill_align/data/processor.py"
click node_src_models "https://github.com/santoshadabala/distill-align-llm/blob/main/src/distill_align/models/loader.py"
click node_src_train "https://github.com/santoshadabala/distill-align-llm/tree/main/src/distill_align/training"
click node_src_serving "https://github.com/santoshadabala/distill-align-llm/tree/main/src/distill_align/serving"
click node_src_tracking "https://github.com/santoshadabala/distill-align-llm/tree/main/src/distill_align/tracking"
click node_run_sft "https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/run_sft.py"
click node_run_dpo "https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/run_dpo.py"
click node_run_bsft "https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/run_bsft.py"
click node_run_full "https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/run_full_pipeline.py"
click node_eval_factuality "https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/eval_factuality.py"
click node_judge_tools "https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/rejudge.py"
click node_benchmarks "https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/generate_benchmark.py"
click node_diagnostics "https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/calibration_ptrue.py"
click node_truthfulqa "https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/eval_truthfulqa.py"
click node_dashboard_app "https://github.com/santoshadabala/distill-align-llm/blob/main/dashboard/app.py"
click node_figure_maker "https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/make_figures.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_data_assets,node_configs toneBlue
class node_src_config,node_src_data,node_src_models,node_src_train,node_src_serving,node_src_tracking toneAmber
class node_run_sft,node_run_dpo,node_run_bsft,node_run_full toneMint
class node_eval_factuality,node_judge_tools,node_benchmarks,node_diagnostics,node_truthfulqa toneRose
class node_dashboard_app,node_figure_maker toneIndigo
class node_openai_anthropic toneTeal
```

## Components

### Assets & Config

Datasets, benchmark prompts, and run configs that feed the research pipeline.

- [Data sets](https://github.com/santoshadabala/distill-align-llm/tree/main/data): jsonl corpora · Raw and curated factuality, trap, instruction, and preference datasets. · `data`
- [Configs](https://github.com/santoshadabala/distill-align-llm/tree/main/configs): yaml runs · Model and experiment configuration files for training sweeps and small/local runs. · `configs`

### Core Library

Reusable Python package for config, data, model, training, serving, and tracking.

- [Run config](https://github.com/santoshadabala/distill-align-llm/blob/main/src/distill_align/config/manager.py): config manager · Loads and manages run configuration and model-specific settings. · `src/distill_align/config/manager.py`
- [Data proc](https://github.com/santoshadabala/distill-align-llm/blob/main/src/distill_align/data/processor.py): dataset processing · Dataset ingestion and transformation logic. · `src/distill_align/data/processor.py`
- [Model loader](https://github.com/santoshadabala/distill-align-llm/blob/main/src/distill_align/models/loader.py): checkpoint loader · Abstracts loading base and fine-tuned models. · `src/distill_align/models/loader.py`
- [Training core](https://github.com/santoshadabala/distill-align-llm/tree/main/src/distill_align/training): sft/dpo/rlhf · Reusable training primitives for supervised, preference, and RLHF-style methods. · `src/distill_align/training`
- [Inference layer](https://github.com/santoshadabala/distill-align-llm/tree/main/src/distill_align/serving): serving/api · Lightweight serving and API engine used for model interaction and diagnostics. · `src/distill_align/serving`
- [Tracking](https://github.com/santoshadabala/distill-align-llm/tree/main/src/distill_align/tracking): telemetry · Run tracking and monitoring support for experiments. · `src/distill_align/tracking`

### Training Jobs

Offline scripts that fine-tune models and generate checkpoints or response logs.

- [SFT run](https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/run_sft.py): training script · Main supervised fine-tuning entry point. · `scripts/run_sft.py`
- [DPO run](https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/run_dpo.py): training script · Preference optimization entry point. · `scripts/run_dpo.py`
- [BSFT run](https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/run_bsft.py): training script · Alternative fine-tuning path used in experiment families. · `scripts/run_bsft.py`
- [Full pipeline](https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/run_full_pipeline.py): orchestrator · End-to-end job that chains training and artifact generation. · `scripts/run_full_pipeline.py`

### Evaluation & Analysis

Post-training scoring, calibration, diagnostics, and benchmark comparisons built from saved artifacts.

- [Factual eval](https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/eval_factuality.py): scored evaluation · Generates factuality metrics from stored model responses. · `scripts/eval_factuality.py`
- [Rejudge & audit](https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/rejudge.py): judge pipeline · Re-scores responses and audits reference answers using external judges. · `scripts/rejudge.py`
- [Benchmarks](https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/generate_benchmark.py): prompt generation · Builds factuality, trap, and eval prompts for benchmark runs. · `scripts/generate_benchmark.py`
- [Diagnostics](https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/calibration_ptrue.py): analysis suite · Calibration, token, attention, probability-mass, and temperature analyses. · `scripts/calibration_ptrue.py`
- [TruthfulQA](https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/eval_truthfulqa.py): benchmark eval · TruthfulQA evaluation and summary generation from saved outputs. · `scripts/eval_truthfulqa.py`

### Presentation

Static dashboard and paper-oriented figure generation over precomputed results.

- [Dashboard](https://github.com/santoshadabala/distill-align-llm/blob/main/dashboard/app.py): streamlit app · Reads precomputed JSON/CSV artifacts and presents comparison views. · `dashboard/app.py`
- [Figures](https://github.com/santoshadabala/distill-align-llm/blob/main/scripts/make_figures.py): reporting · Builds paper figures and summary plots from stored artifacts. · `scripts/make_figures.py`

### External Judges

Remote LLM APIs used for re-scoring and reference auditing.

- **LLM APIs**: remote judges · External OpenAI and Anthropic models used for scoring and auditing.

## Connections

- Data sets → Data proc: ingest · Datasets are transformed into training and evaluation-ready examples.
- Configs → Run config: configure · YAML configs define model and experiment settings.
- Run config → SFT run: drives · Training jobs load run settings from the config manager.
- Run config → DPO run: drives · Training jobs load run settings from the config manager.
- Run config → BSFT run: drives · Training jobs load run settings from the config manager.
- Data proc → SFT run: feeds · Processed datasets are consumed by supervised fine-tuning.
- Data proc → DPO run: feeds · Processed datasets are consumed by preference optimization.
- Model loader → SFT run: loads · Training jobs load base and adapted models through the loader boundary.
- Model loader → DPO run: loads · Training jobs load base and adapted models through the loader boundary.
- Training core → SFT run: implements · SFT script uses shared training primitives.
- Training core → DPO run: implements · DPO script uses shared training primitives.
- Training core → BSFT run: implements · BSFT script uses shared training primitives.
- SFT run → Full pipeline: stage · Full pipeline orchestrates the supervised fine-tuning stage.
- DPO run → Full pipeline: stage · Full pipeline orchestrates the preference-optimization stage.
- Full pipeline → Factual eval: outputs · Pipeline artifacts are passed into evaluation jobs.
- SFT run → Factual eval: produces · Fine-tuned checkpoints and responses are later evaluated.
- DPO run → Factual eval: produces · Fine-tuned checkpoints and responses are later evaluated.
- Benchmarks → Factual eval: inputs · Benchmark prompt generation supplies evaluation sets.
- Benchmarks → TruthfulQA: inputs · Generated prompts support benchmark-specific evaluation.
- Factual eval → LLM APIs: judges · Factuality scoring can be delegated to external judge models.
- Rejudge & audit → LLM APIs: calls · Rejudging and reference audits use external API-backed models.
- Rejudge & audit → Factual eval: rescoring · Stored responses are rescored to refresh summary metrics.
- Diagnostics → Factual eval: analyzes · Calibration and token-level diagnostics consume saved evaluation artifacts.
- TruthfulQA → Dashboard: publishes · Benchmark summaries are surfaced in the dashboard.
- Factual eval → Dashboard: publishes · Evaluation outputs feed the presentation layer.
- Figures → Dashboard: shares artifacts · Figures are built from the same precomputed results used by the dashboard.
- Inference layer → Diagnostics: supports · Serving and inference utilities help generate logits and response traces.
- Tracking → Full pipeline: records · Experiment tracking captures job metadata and outputs.
