How shatadip/tutorbugbench works
TutorBugBench is an execution-verified benchmark for whether LLMs can identify bugs and give non-revealing hints in introductory Python submissions. Its workflow defines labelled exercises, validates labels by running tests, queries a selected model for each submission, and records per-run predictions and metrics. A separate aggregation stage compares completed runs and produces summary data and paper tables. The README documents the dataset and reporting capabilities; the sampled source confirms the evaluation and aggregation wiring, but not the internal implementation of the dataset or paper.
Main entry points: run_eval.py Evaluation runner · analyze.py Run aggregation · validate.py Label validation · results_tables.tex Paper result tables
Components 12
Benchmark data
- Exercises and mutations
bench/items.py
Evaluation
- Label validation
validate.py - Evaluation runner
run_eval.py - Submission test execution
run_eval.py - Per-item results
results - Run metrics
results
Analysis and reporting
- Run aggregation
analyze.py - Aggregate summary
results/summary.json - Paper result tables
paper/results_tables.tex - Research paper
paper/main.tex
Model providers
- LLM provider APIs
Other components
- Benchmark user
Connections 14
- Benchmark user → Label validation: checks labels
- Label validation → Exercises and mutations: loads items
- Label validation → Submission test execution: runs tests
- Benchmark user → Evaluation runner: starts evaluation
- Evaluation runner → Exercises and mutations: loads items
- Evaluation runner → Submission test execution: executes tests
- Evaluation runner → LLM provider APIs: queries model
- Evaluation runner → Per-item results: writes predictions
- Evaluation runner → Run metrics: writes metrics
- Benchmark user → Run aggregation: aggregates runs
- Run aggregation → Per-item results: reads completed runs
- Run aggregation → Aggregate summary: writes summary
- Run aggregation → Paper result tables: writes tables
- Paper result tables → Research paper: supplies results