# mqaryouti89/minicheck-reproduction architecture

> Architecture diagram of the GitHub repository mqaryouti89/minicheck-reproduction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mqaryouti89/minicheck-reproduction
- Repository: https://github.com/mqaryouti89/minicheck-reproduction

## Overview

This repository is a research evaluation workflow: it reproduces MiniCheck scores on the paper’s benchmark, evaluates a confidence-based RoBERTa-to-Flan-T5 cascade, analyzes errors, and assembles checked report results. The main path is checker inference → saved predictions → metrics and cascade evaluation → error analysis and final tables. The graph follows the README and sampled scripts; unsampled implementation details are omitted where their call relationships cannot be verified.

Main entry points: [`scripts/final_results.py`](https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/final_results.py) (Final Tables), [`results/kaggle-t4/minicheck_results`](https://github.com/mqaryouti89/minicheck-reproduction/tree/main/results/kaggle-t4/minicheck_results) (Saved Predictions), [`results/phase4/selected_thresholds.json`](https://github.com/mqaryouti89/minicheck-reproduction/blob/main/results/phase4/selected_thresholds.json) (Frozen Threshold), [`scripts/run_checker.py`](https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/run_checker.py) (MiniCheck Checkers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_scoring["Checker Scoring"]
  node_checkers["MiniCheck Checkers<br/>[run_checker.py]"]
  node_predictions[("Saved Predictions")]
end

subgraph group_cascade["Cascade Study"]
  node_plan["Cascade Plan<br/>[phase4.json]"]
  node_thresholds["Frozen Threshold"]
  node_selection["Threshold Selection<br/>[phase4.py]"]
  node_cascade_runner["Cascade Inference<br/>[run_cascade.py]"]
  node_cascade_results[("Cascade Results")]
  node_evaluation["Cascade Evaluation<br/>[phase4.py]"]
  node_phase4_results[("Phase 4 Results")]
end

subgraph group_analysis["Evaluation Analysis"]
  node_metrics["Checker Metrics<br/>[metrics.py]"]
  node_error_analysis["Error Analysis<br/>[error_analysis.py]"]
  node_phase5_results[("Phase 5 Results")]
end

subgraph group_reporting["Results Reporting"]
  node_final_results["Final Tables<br/>[final_results.py]"]
  node_final_store[("Final Report Data")]
end

node_researcher(("Researcher"))
node_benchmark[("LLM-AggreFact")]

node_researcher -->|"runs scoring"| node_checkers
node_benchmark -->|"provides claims"| node_checkers
node_checkers -->|"writes predictions"| node_predictions
node_predictions -->|"supplies scores"| node_metrics
node_researcher -->|"starts study"| node_selection
node_plan -->|"constrains selection"| node_selection
node_predictions -.->|"supplies dev scores"| node_selection
node_selection -->|"freezes threshold"| node_thresholds
node_thresholds -->|"sets defer rule"| node_cascade_runner
node_checkers -->|"provides loading functions"| node_cascade_runner
node_benchmark -->|"provides claims"| node_cascade_runner
node_cascade_runner -->|"writes cascade scores"| node_cascade_results
node_cascade_results -.->|"supplies measured run"| node_evaluation
node_thresholds -->|"sets test threshold"| node_evaluation
node_evaluation -->|"writes evaluation"| node_phase4_results
node_predictions -->|"supplies predictions"| node_error_analysis
node_thresholds -->|"sets cascade rule"| node_error_analysis
node_phase4_results -->|"supplies cascade baseline"| node_error_analysis
node_error_analysis -->|"writes analyses"| node_phase5_results
node_researcher -->|"builds final report"| node_final_results
node_metrics -.->|"contributes summaries"| node_final_results
node_phase4_results -->|"contributes evaluation"| node_final_results
node_phase5_results -->|"contributes analysis"| node_final_results
node_final_results -->|"writes tables"| node_final_store

click node_checkers "https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/run_checker.py"
click node_predictions "https://github.com/mqaryouti89/minicheck-reproduction/tree/main/results/kaggle-t4/minicheck_results"
click node_metrics "https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/metrics.py"
click node_plan "https://github.com/mqaryouti89/minicheck-reproduction/blob/main/configs/phase4.json"
click node_thresholds "https://github.com/mqaryouti89/minicheck-reproduction/blob/main/results/phase4/selected_thresholds.json"
click node_selection "https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/phase4.py"
click node_cascade_runner "https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/run_cascade.py"
click node_cascade_results "https://github.com/mqaryouti89/minicheck-reproduction/tree/main/results/kaggle-t4-cascade/minicheck_results"
click node_evaluation "https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/phase4.py"
click node_error_analysis "https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/error_analysis.py"
click node_phase4_results "https://github.com/mqaryouti89/minicheck-reproduction/tree/main/results/phase4"
click node_phase5_results "https://github.com/mqaryouti89/minicheck-reproduction/tree/main/results/phase5"
click node_final_results "https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/final_results.py"
click node_final_store "https://github.com/mqaryouti89/minicheck-reproduction/tree/main/results/final"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_checkers,node_predictions toneBlue
class node_plan,node_thresholds,node_selection,node_cascade_runner,node_cascade_results,node_evaluation,node_phase4_results,node_benchmark toneAmber
class node_metrics,node_error_analysis,node_phase5_results toneMint
class node_final_results,node_final_store toneRose
class node_researcher toneIndigo
```

## Components

### Checker Scoring

- [MiniCheck Checkers](https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/run_checker.py): `scripts/run_checker.py`
- [Saved Predictions](https://github.com/mqaryouti89/minicheck-reproduction/tree/main/results/kaggle-t4/minicheck_results): `results/kaggle-t4/minicheck_results`

### Cascade Study

- [Cascade Plan](https://github.com/mqaryouti89/minicheck-reproduction/blob/main/configs/phase4.json): `configs/phase4.json`
- [Frozen Threshold](https://github.com/mqaryouti89/minicheck-reproduction/blob/main/results/phase4/selected_thresholds.json): `results/phase4/selected_thresholds.json`
- [Threshold Selection](https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/phase4.py): `scripts/phase4.py`
- [Cascade Inference](https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/run_cascade.py): `scripts/run_cascade.py`
- [Cascade Results](https://github.com/mqaryouti89/minicheck-reproduction/tree/main/results/kaggle-t4-cascade/minicheck_results): `results/kaggle-t4-cascade/minicheck_results`
- [Cascade Evaluation](https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/phase4.py): `scripts/phase4.py`
- [Phase 4 Results](https://github.com/mqaryouti89/minicheck-reproduction/tree/main/results/phase4): `results/phase4`

### Evaluation Analysis

- [Checker Metrics](https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/metrics.py): `scripts/metrics.py`
- [Error Analysis](https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/error_analysis.py): `scripts/error_analysis.py`
- [Phase 5 Results](https://github.com/mqaryouti89/minicheck-reproduction/tree/main/results/phase5): `results/phase5`

### Results Reporting

- [Final Tables](https://github.com/mqaryouti89/minicheck-reproduction/blob/main/scripts/final_results.py): `scripts/final_results.py`
- [Final Report Data](https://github.com/mqaryouti89/minicheck-reproduction/tree/main/results/final): `results/final`

### Other components

- **Researcher**
- **LLM-AggreFact**

## Connections

- Researcher → MiniCheck Checkers: runs scoring (evidence: `README.md`)
- LLM-AggreFact → MiniCheck Checkers: provides claims (evidence: `README.md`)
- MiniCheck Checkers → Saved Predictions: writes predictions (evidence: `scripts/run_checker.py`)
- Saved Predictions → Checker Metrics: supplies scores (evidence: `scripts/metrics.py`)
- Researcher → Threshold Selection: starts study (evidence: `README.md`)
- Cascade Plan → Threshold Selection: constrains selection (evidence: `README.md`)
- Saved Predictions → Threshold Selection: supplies dev scores (evidence: `scripts/phase4.py`)
- Threshold Selection → Frozen Threshold: freezes threshold (evidence: `README.md`)
- Frozen Threshold → Cascade Inference: sets defer rule (evidence: `scripts/run_cascade.py`)
- MiniCheck Checkers → Cascade Inference: provides loading functions (evidence: `scripts/run_cascade.py`)
- LLM-AggreFact → Cascade Inference: provides claims (evidence: `README.md`)
- Cascade Inference → Cascade Results: writes cascade scores (evidence: `scripts/run_cascade.py`)
- Cascade Results → Cascade Evaluation: supplies measured run (evidence: `README.md`)
- Frozen Threshold → Cascade Evaluation: sets test threshold (evidence: `README.md`)
- Cascade Evaluation → Phase 4 Results: writes evaluation (evidence: `README.md`)
- Saved Predictions → Error Analysis: supplies predictions (evidence: `scripts/error_analysis.py`)
- Frozen Threshold → Error Analysis: sets cascade rule (evidence: `scripts/error_analysis.py`)
- Phase 4 Results → Error Analysis: supplies cascade baseline (evidence: `scripts/error_analysis.py`)
- Error Analysis → Phase 5 Results: writes analyses (evidence: `scripts/error_analysis.py`)
- Researcher → Final Tables: builds final report (evidence: `README.md`)
- Checker Metrics → Final Tables: contributes summaries (evidence: `scripts/final_results.py`)
- Phase 4 Results → Final Tables: contributes evaluation (evidence: `scripts/final_results.py`)
- Phase 5 Results → Final Tables: contributes analysis (evidence: `scripts/final_results.py`)
- Final Tables → Final Report Data: writes tables (evidence: `scripts/final_results.py`)
