# kiangyeeo/parametric-faithfulness architecture

> Architecture diagram of the GitHub repository kiangyeeo/parametric-faithfulness, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kiangyeeo/parametric-faithfulness
- Repository: https://github.com/kiangyeeo/parametric-faithfulness

## Overview

This repository is a research reproduction and extension of FUR, investigating whether a language model’s reasoning steps are faithful to its answers. The main workflow generates CoT and no-CoT baselines, unlearns reasoning steps, evaluates post-unlearning behavior, and computes faithfulness, efficacy, and specificity metrics. Separate experiments test injected mistakes, regularization-weight sweeps, alternative objectives, and word-type selection. The README documents these capabilities; the graph uses sampled source for the reproduction runner, unlearning, mistake evaluation, and metric calculation, and does not infer wiring for unsampled experiment code.

Main entry points: [`repro/run_repro.py`](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/repro/run_repro.py) (Reproduction runner), [`add_mistakes/mistakes_repro.py`](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/add_mistakes/mistakes_repro.py) (Mistake baseline), [`models.py`](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/models.py) (Model loading), [`unlearn.py`](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/unlearn.py) (Step unlearning)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data and Models"]
  node_qa_data["QA datasets"]
  node_loaders["Dataset handlers<br/>[dataload.py]"]
  node_model_loader["Model loading<br/>[models.py]"]
end

subgraph group_repro["FUR Reproduction"]
  node_runner["Reproduction runner<br/>[run_repro.py]"]
  node_unlearning["Step unlearning<br/>[unlearn.py]"]
  node_prompts["CoT baselines"]
  node_results["Unlearning results"]
  node_eval["Answer evaluation<br/>[evaluate.py]"]
end

subgraph group_extensions["Research Extensions"]
  node_mistake_injection["Mistake baseline<br/>[mistakes_repro.py]"]
  node_mistake_results["Injected mistakes"]
  node_mistake_stats["Mistake evaluations"]
  node_lambda_sweep["Lambda experiments"]
  node_wordtype_unlearn["Word-type unlearning"]
end

subgraph group_analysis["Evaluation and Analysis"]
  node_metrics["FUR metrics<br/>[compute_metrics.py]"]
  node_mistake_metrics["Mistake metrics"]
  node_lambda_analysis["Lambda analysis"]
  node_wordtype_analysis["Word-type analysis"]
  node_mmlu["MMLU evaluation"]
end

node_researcher(("Researcher"))
node_hf_models[("Hugging Face models")]

node_researcher -->|"starts runs"| node_runner
node_researcher -->|"starts baseline"| node_mistake_injection
node_qa_data -->|"provides examples"| node_loaders
node_runner -->|"loads data"| node_loaders
node_runner -->|"loads tokenizer"| node_model_loader
node_model_loader -.->|"loads weights"| node_hf_models
node_runner -->|"runs unlearning"| node_unlearning
node_unlearning -->|"uses model"| node_model_loader
node_unlearning -->|"evaluates answers"| node_eval
node_runner -->|"reads CoT inputs"| node_prompts
node_runner -->|"writes outcomes"| node_results
node_mistake_injection -->|"loads model"| node_model_loader
node_mistake_injection -->|"scores answers"| node_eval
node_mistake_injection -->|"reads mistakes"| node_mistake_results
node_mistake_injection -->|"writes scores"| node_mistake_stats
node_metrics -->|"reads outcomes"| node_results
node_mistake_metrics -->|"reads injections"| node_mistake_results
node_mistake_metrics -->|"reads scores"| node_mistake_stats
node_lambda_sweep -->|"configures runs"| node_runner
node_lambda_analysis -->|"summarizes results"| node_lambda_sweep
node_wordtype_unlearn -->|"loads model"| node_model_loader
node_wordtype_analysis -->|"analyzes outcomes"| node_results
node_runner -->|"writes evaluations"| node_mmlu

click node_qa_data "https://github.com/kiangyeeo/parametric-faithfulness/tree/main/data"
click node_loaders "https://github.com/kiangyeeo/parametric-faithfulness/blob/main/dataload.py"
click node_model_loader "https://github.com/kiangyeeo/parametric-faithfulness/blob/main/models.py"
click node_runner "https://github.com/kiangyeeo/parametric-faithfulness/blob/main/repro/run_repro.py"
click node_unlearning "https://github.com/kiangyeeo/parametric-faithfulness/blob/main/unlearn.py"
click node_prompts "https://github.com/kiangyeeo/parametric-faithfulness/tree/main/final_cot"
click node_results "https://github.com/kiangyeeo/parametric-faithfulness/tree/main/final_results"
click node_eval "https://github.com/kiangyeeo/parametric-faithfulness/blob/main/evaluate.py"
click node_mistake_injection "https://github.com/kiangyeeo/parametric-faithfulness/blob/main/add_mistakes/mistakes_repro.py"
click node_mistake_results "https://github.com/kiangyeeo/parametric-faithfulness/tree/main/add_mistakes/mistake_results"
click node_mistake_stats "https://github.com/kiangyeeo/parametric-faithfulness/tree/main/add_mistakes/mistake_stats"
click node_lambda_sweep "https://github.com/kiangyeeo/parametric-faithfulness/tree/main/lambda"
click node_wordtype_unlearn "https://github.com/kiangyeeo/parametric-faithfulness/blob/main/word_type_select/unlearn_for_more_kinds_of_words.py"
click node_metrics "https://github.com/kiangyeeo/parametric-faithfulness/blob/main/repro/compute_metrics.py"
click node_mistake_metrics "https://github.com/kiangyeeo/parametric-faithfulness/blob/main/add_mistakes/compute_mistake_baseline.py"
click node_lambda_analysis "https://github.com/kiangyeeo/parametric-faithfulness/blob/main/lambda/analyze_lambda_results.py"
click node_wordtype_analysis "https://github.com/kiangyeeo/parametric-faithfulness/blob/main/word_type_select/word_type_analysis.py"
click node_mmlu "https://github.com/kiangyeeo/parametric-faithfulness/tree/main/mmlu_results"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_qa_data,node_loaders,node_model_loader toneBlue
class node_runner,node_unlearning,node_prompts,node_results,node_eval,node_hf_models toneAmber
class node_mistake_injection,node_mistake_results,node_mistake_stats,node_lambda_sweep,node_wordtype_unlearn toneMint
class node_metrics,node_mistake_metrics,node_lambda_analysis,node_wordtype_analysis,node_mmlu toneRose
class node_researcher toneIndigo
```

## Components

### Data and Models

- [QA datasets](https://github.com/kiangyeeo/parametric-faithfulness/tree/main/data): `data`
- [Dataset handlers](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/dataload.py): `dataload.py`
- [Model loading](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/models.py): `models.py`

### FUR Reproduction

- [Reproduction runner](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/repro/run_repro.py): `repro/run_repro.py`
- [Step unlearning](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/unlearn.py): `unlearn.py`
- [CoT baselines](https://github.com/kiangyeeo/parametric-faithfulness/tree/main/final_cot): `final_cot`
- [Unlearning results](https://github.com/kiangyeeo/parametric-faithfulness/tree/main/final_results): `final_results`
- [Answer evaluation](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/evaluate.py): `evaluate.py`

### Research Extensions

- [Mistake baseline](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/add_mistakes/mistakes_repro.py): `add_mistakes/mistakes_repro.py`
- [Injected mistakes](https://github.com/kiangyeeo/parametric-faithfulness/tree/main/add_mistakes/mistake_results): `add_mistakes/mistake_results`
- [Mistake evaluations](https://github.com/kiangyeeo/parametric-faithfulness/tree/main/add_mistakes/mistake_stats): `add_mistakes/mistake_stats`
- [Lambda experiments](https://github.com/kiangyeeo/parametric-faithfulness/tree/main/lambda): `lambda`
- [Word-type unlearning](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/word_type_select/unlearn_for_more_kinds_of_words.py): `word_type_select/unlearn_for_more_kinds_of_words.py`

### Evaluation and Analysis

- [FUR metrics](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/repro/compute_metrics.py): `repro/compute_metrics.py`
- [Mistake metrics](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/add_mistakes/compute_mistake_baseline.py): `add_mistakes/compute_mistake_baseline.py`
- [Lambda analysis](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/lambda/analyze_lambda_results.py): `lambda/analyze_lambda_results.py`
- [Word-type analysis](https://github.com/kiangyeeo/parametric-faithfulness/blob/main/word_type_select/word_type_analysis.py): `word_type_select/word_type_analysis.py`
- [MMLU evaluation](https://github.com/kiangyeeo/parametric-faithfulness/tree/main/mmlu_results): `mmlu_results`

### Other components

- **Researcher**
- **Hugging Face models**

## Connections

- Researcher → Reproduction runner: starts runs
- Researcher → Mistake baseline: starts baseline
- QA datasets → Dataset handlers: provides examples
- Reproduction runner → Dataset handlers: loads data
- Reproduction runner → Model loading: loads tokenizer
- Model loading → Hugging Face models: loads weights
- Reproduction runner → Step unlearning: runs unlearning
- Step unlearning → Model loading: uses model
- Step unlearning → Answer evaluation: evaluates answers
- Reproduction runner → CoT baselines: reads CoT inputs
- Reproduction runner → Unlearning results: writes outcomes
- Mistake baseline → Model loading: loads model
- Mistake baseline → Answer evaluation: scores answers
- Mistake baseline → Injected mistakes: reads mistakes
- Mistake baseline → Mistake evaluations: writes scores
- FUR metrics → Unlearning results: reads outcomes
- Mistake metrics → Injected mistakes: reads injections
- Mistake metrics → Mistake evaluations: reads scores
- Lambda experiments → Reproduction runner: configures runs
- Lambda analysis → Lambda experiments: summarizes results
- Word-type unlearning → Model loading: loads model
- Word-type analysis → Unlearning results: analyzes outcomes
- Reproduction runner → MMLU evaluation: writes evaluations