# shatadip/tutorbugbench architecture

> Architecture diagram of the GitHub repository shatadip/tutorbugbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shatadip/tutorbugbench
- Repository: https://github.com/shatadip/tutorbugbench

## Overview

TutorBugBench is an execution-verified benchmark for whether LLMs can identify bugs and give non-revealing hints in introductory Python submissions. Its workflow defines labelled exercises, validates labels by running tests, queries a selected model for each submission, and records per-run predictions and metrics. A separate aggregation stage compares completed runs and produces summary data and paper tables. The README documents the dataset and reporting capabilities; the sampled source confirms the evaluation and aggregation wiring, but not the internal implementation of the dataset or paper.

Main entry points: [`run_eval.py`](https://github.com/shatadip/tutorbugbench/blob/main/run_eval.py) (Evaluation runner), [`analyze.py`](https://github.com/shatadip/tutorbugbench/blob/main/analyze.py) (Run aggregation), [`validate.py`](https://github.com/shatadip/tutorbugbench/blob/main/validate.py) (Label validation), [`paper/results_tables.tex`](https://github.com/shatadip/tutorbugbench/blob/main/paper/results_tables.tex) (Paper result tables)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_benchmark["Benchmark data"]
  node_items["Exercises and mutations<br/>[items.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_validator["Label validation<br/>[validate.py]"]
  node_runner["Evaluation runner<br/>[run_eval.py]"]
  node_execution_tests["Submission test execution<br/>[run_eval.py]"]
  node_raw_results[("Per-item results")]
  node_run_metrics["Run metrics"]
end

subgraph group_reporting["Analysis and reporting"]
  node_aggregator["Run aggregation<br/>[analyze.py]"]
  node_summary["Aggregate summary<br/>[summary.json]"]
  node_tables["Paper result tables<br/>[results_tables.tex]"]
  node_paper["Research paper<br/>[main.tex]"]
end

subgraph group_integrations["Model providers"]
  node_provider_services{{"LLM provider APIs"}}
end

node_researcher(("Benchmark user"))

node_researcher -->|"checks labels"| node_validator
node_validator -->|"loads items"| node_items
node_validator -->|"runs tests"| node_execution_tests
node_researcher -->|"starts evaluation"| node_runner
node_runner -->|"loads items"| node_items
node_runner -->|"executes tests"| node_execution_tests
node_runner -->|"queries model"| node_provider_services
node_runner -->|"writes predictions"| node_raw_results
node_runner -->|"writes metrics"| node_run_metrics
node_researcher -->|"aggregates runs"| node_aggregator
node_aggregator -->|"reads completed runs"| node_raw_results
node_aggregator -->|"writes summary"| node_summary
node_aggregator -->|"writes tables"| node_tables
node_tables -->|"supplies results"| node_paper

click node_items "https://github.com/shatadip/tutorbugbench/blob/main/bench/items.py"
click node_validator "https://github.com/shatadip/tutorbugbench/blob/main/validate.py"
click node_runner "https://github.com/shatadip/tutorbugbench/blob/main/run_eval.py"
click node_execution_tests "https://github.com/shatadip/tutorbugbench/blob/main/run_eval.py"
click node_raw_results "https://github.com/shatadip/tutorbugbench/tree/main/results"
click node_run_metrics "https://github.com/shatadip/tutorbugbench/tree/main/results"
click node_aggregator "https://github.com/shatadip/tutorbugbench/blob/main/analyze.py"
click node_summary "https://github.com/shatadip/tutorbugbench/blob/main/results/summary.json"
click node_tables "https://github.com/shatadip/tutorbugbench/blob/main/paper/results_tables.tex"
click node_paper "https://github.com/shatadip/tutorbugbench/blob/main/paper/main.tex"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_items,node_researcher toneBlue
class node_validator,node_runner,node_execution_tests,node_raw_results,node_run_metrics toneAmber
class node_aggregator,node_summary,node_tables,node_paper toneMint
class node_provider_services toneRose
```

## Components

### Benchmark data

- [Exercises and mutations](https://github.com/shatadip/tutorbugbench/blob/main/bench/items.py): `bench/items.py`

### Evaluation

- [Label validation](https://github.com/shatadip/tutorbugbench/blob/main/validate.py): `validate.py`
- [Evaluation runner](https://github.com/shatadip/tutorbugbench/blob/main/run_eval.py): `run_eval.py`
- [Submission test execution](https://github.com/shatadip/tutorbugbench/blob/main/run_eval.py): `run_eval.py`
- [Per-item results](https://github.com/shatadip/tutorbugbench/tree/main/results): `results`
- [Run metrics](https://github.com/shatadip/tutorbugbench/tree/main/results): `results`

### Analysis and reporting

- [Run aggregation](https://github.com/shatadip/tutorbugbench/blob/main/analyze.py): `analyze.py`
- [Aggregate summary](https://github.com/shatadip/tutorbugbench/blob/main/results/summary.json): `results/summary.json`
- [Paper result tables](https://github.com/shatadip/tutorbugbench/blob/main/paper/results_tables.tex): `paper/results_tables.tex`
- [Research paper](https://github.com/shatadip/tutorbugbench/blob/main/paper/main.tex): `paper/main.tex`

### Model providers

- **LLM provider APIs**

### Other components

- **Benchmark user**

## Connections

- Benchmark user → Label validation: checks labels
- Label validation → Exercises and mutations: loads items
- Label validation → Submission test execution: runs tests
- Benchmark user → Evaluation runner: starts evaluation
- Evaluation runner → Exercises and mutations: loads items
- Evaluation runner → Submission test execution: executes tests
- Evaluation runner → LLM provider APIs: queries model
- Evaluation runner → Per-item results: writes predictions
- Evaluation runner → Run metrics: writes metrics
- Benchmark user → Run aggregation: aggregates runs
- Run aggregation → Per-item results: reads completed runs
- Run aggregation → Aggregate summary: writes summary
- Run aggregation → Paper result tables: writes tables
- Paper result tables → Research paper: supplies results