rifoagenadi/sycophancy

How rifoagenadi/sycophancy works

This repository develops and evaluates probes for sycophancy and truthfulness in language-model activations, with a separate dialogue-based steering and uncertainty-quantification workstream. The main workflow prepares labeled TruthfulQA examples, extracts MHA/MLP/residual activations, trains probes, and applies probe directions during inference. Additional branches generate disagreement dialogues, steer or ablate model responses, and evaluate or visualize results. Edges reflect the README or shown source/index references; unsampled subsystem internals and some runtime wiring are intentionally left unspecified.

Main entry points: utils.py Model and data loading · train.py Trained probes · metrics.py Metrics · dialogue.py Disagreement dialogues

Components 27

Probe workflow

Labeled examples probe/probe_data_utils.py
Model and data loading utils.py
Activation extraction probe/extract_activation.py
Probe training probe/train.py
Trained probes probe/train.py
Projection statistics probe/compute_proj_std.py
MHA intervention inference_mha.py
MLP intervention inference_mlp.py
Residual intervention inference_residual.py

Dialogue and steering

Disagreement dialogues uq/dialogue.py
Steering dialogues extension/dialogue.py
Dialogue activations extension/extract_activations.py
Steering and ablation extension/steer.py

Evaluation and analysis

Response judging extension/judge.py
Metrics extension/metrics.py
Probe accuracy plots extension/diagram/probe_accuracy_plot.py
Disagreement plots extension/diagram/disagreement_type_plot.py
Steering plots extension/diagram/steering_generalization_plot.py

Uncertainty quantification

UQ dialogue uq/dialogue.py
UQ judging uq/judge.py
UQ labeling uq/label.py
UQ runner uq/run_uqlm.py
UQ visualization uq/visualize.py

Other components

Researcher
TruthfulQA data
Model provider
Model responses

Connections 32

Researcher → Probe training: trains probes README.md
TruthfulQA data → Labeled examples: provides examples probe/probe_data_utils.py
TruthfulQA data → Model and data loading: loads test data utils.py
Model provider → Model and data loading: provides models utils.py
Labeled examples → Probe training: labels data probe/train.py
Model and data loading → Probe training: loads model probe/train.py
Probe training → Activation extraction: extracts activations probe/train.py
Probe training → Trained probes: saves weights README.md
Trained probes → Projection statistics: projects directions probe/compute_proj_std.py
Trained probes → MHA intervention: loads directions inference_mha.py
Trained probes → MLP intervention: loads directions inference_mlp.py
Trained probes → Residual intervention: loads directions inference_residual.py
Model and data loading → MHA intervention: loads model and data inference_mha.py
Model and data loading → MLP intervention: loads model and data inference_mlp.py
Model and data loading → Residual intervention: loads model and data inference_residual.py
MHA intervention → Model responses: writes predictions README.md
MLP intervention → Model responses: writes predictions inference_mlp.py
Residual intervention → Model responses: writes predictions inference_residual.py
TruthfulQA data → Disagreement dialogues: loads questions uq/dialogue.py
Disagreement dialogues → Model responses: writes dialogues uq/dialogue.py
TruthfulQA data → Dialogue activations: builds dialogue samples extension/extract_activations.py
Dialogue activations → Steering and ablation: supplies probe inputs extension/steer.py
Disagreement dialogues → Steering and ablation: dialogue reference extension/steer.py
Researcher → Response judging: evaluates responses
Response judging → Metrics: provides judgments
Metrics → Probe accuracy plots: plots accuracy
Metrics → Disagreement plots: plots disagreement
Metrics → Steering plots: plots generalization
Researcher → UQ dialogue: runs UQ dialogue
UQ dialogue → UQ judging: judgment workflow
UQ judging → UQ labeling: labeling workflow
UQ runner → UQ visualization: analysis workflow

More Jupyter Notebook diagrams: bowang-lab/scgpt · nvlabs/sionna · superlinked/superlinked · sajal2692/data-science-portfolio · nousresearch/hermes-function-calling

Drawn by GitDiagram on November 10, 2026 from the repository's file tree, README and source files. Read as Markdown ·