How mathura-snth/math-llm-verifier works
The active product is a reproducible benchmark generator for evaluating whether language models detect logical errors in theorem hypotheses. A user runs the benchmark, which selects theorems and error types, generates and formally validates synthetic copies, and writes labeled cases to JSON. The formal model comprises theorem requirements, hypothesis text, and implication rules. The repository also preserves an earlier French-language implementation and a Tableau exporter. The main workflow is evidenced by sampled active source; the archive is shown as a separate historical branch, not part of the active pipeline.
Main entry points: run.py Dataset runner · generate_copy.py Copy generator · generer_copie.py Historical generator · run_benchmark.py Historical runner
Components 14
Active benchmark
- Dataset runner
eng_ver/run.py - Copy generator
eng_ver/generate_copy.py
Formal logic
- Theorem catalog
eng_ver/theorems.py - Hypothesis dictionary
eng_ver/hypotheses.py - Implication engine
eng_ver/implications.py
Archived version
- Historical runner
archives/run_benchmark.py - Historical generator
archives/generer_copie.py - French theorem catalog
archives/theoremes.py - French hypothesis dictionary
archives/hypotheses.py - Historical implication engine
archives/implications.py - Table exporter
archives/exporter_tableau.py
Benchmark data
- Generated cases
eng_ver/benchmark_data.json - Historical dataset
archives/benchmark_data.json
Other components
- Benchmark user
Connections 19
- Benchmark user → Dataset runner: runs
README.md - Dataset runner → Theorem catalog: selects from
eng_ver/run.py - Dataset runner → Copy generator: generates with
eng_ver/run.py - Copy generator → Theorem catalog: reads requirements
eng_ver/generate_copy.py - Copy generator → Hypothesis dictionary: uses text
eng_ver/generate_copy.py - Copy generator → Implication engine: checks implications
eng_ver/generate_copy.py - Theorem catalog → Hypothesis dictionary: resolves identifiers
eng_ver/theorems.py - Implication engine → Hypothesis dictionary: uses identifiers
eng_ver/implications.py - Dataset runner → Generated cases: writes cases
eng_ver/run.py - Historical runner → French theorem catalog: selects from
archives/run_benchmark.py - Historical runner → Historical generator: generates with
archives/run_benchmark.py - Historical generator → French theorem catalog: reads requirements
archives/generer_copie.py - Historical generator → French hypothesis dictionary: uses text
archives/generer_copie.py - Historical generator → Historical implication engine: checks implications
archives/generer_copie.py - French theorem catalog → French hypothesis dictionary: resolves identifiers
archives/theoremes.py - Historical implication engine → French hypothesis dictionary: uses identifiers
archives/implications.py - Historical runner → Historical dataset: writes cases
archives/run_benchmark.py - Table exporter → Historical dataset: loads cases
archives/exporter_tableau.py - Benchmark user → Table exporter: exports tables
README.md