How opendatalab/provergen works
Project type and purpose
• Project type: A command-line, data-generation and evaluation framework (open-source tool) for creating logical reasoning problems (ProverGen).
• Purpose:
• Generate first-order logic (FOL) “skeletons” using a symbolic prover (Prover9)
• Translate skeletons into natural language via large language models (LLMs)
• Produce final FOL reasoning problems (with augmentations)
• Evaluate LLM performance against various datasets
File structure
• Top-level scripts:
• logic_skeleton_generator.py → invokes Prover9 to produce logic skeletons
• logic_skeleton_translator.py → uses LLMs to translate skeletons to nat-lang
• fol_problem_generator.py → generates FOL problems with distractions/augmentations
• evaluation.py → runs LLMs on generated datasets
• result/metrics.py → computes metrics on evaluation outputs
• Data directories:
• data/ → rule pools, name lists, example translations
• logic_data/ → skeleton outputs and few-shot examples for each benchmark
• training_data/ → prebuilt datasets (folio, proofwriter…)
• Utilities under utils/:
• llms_interface.py, prover9_interface.py → abstract external services
• augmentor.py → data augmentation
• logic_generator/ & logic_translator/ → internal pipeline components
• Configuration: requirements.txt, .gitignore
README for additional insights
• Describes CLI parameters for each stage (num, mode, start/end, noise flags)
• Relies on external services: Prover9 binary + LLM API (HuggingFace or local server)
a. Main components:
1. Data Storage
• Raw resources (data/, training_data/)
• Intermediate output (logic_data/, translated_data/)
• Final problem sets + evaluation results (outputs/, result/)
2. Logic Skeleton Generator
• Script: logic_skeleton_generator.py
• Calls Prover9 via utils/prover9_interface.py
3. Logic Skeleton Translator
• Script: logic_skeleton_translator.py
• Calls LLMs via utils/llms_interface.py
4. FOL Problem Generator
• Script: fol_problem_generator.py
• Uses augmentor.py for noise and paraphrase
5. Evaluation Module
• Script: evaluation.py
• Metrics computed by result/metrics.py
6. External Services
• Prover9 binary (symbolic prover)
• LLM service (Hugging Face or local)
b. Component interactions (data flow):
Data resources → Logic Skeleton Generator → logic_data/
logic_data/ + LLM Interface → Logic Skeleton Translator → translated_data/
translated_data/ + augmentor → FOL Problem Generator → final_data/
final_data/ + LLM Interface → Evaluation → result/ → metrics computation
c. Architectural patterns / principles:
• Modular pipeline – each stage is an independent, reusable script
• Adapter pattern – utils/llms_interface.py and prover9_interface.py abstract external services
• Data-driven – JSON files passed between stages
d. Key technologies / frameworks:
• Python 3, JSON, Conda environment
• Prover9 (symbolic prover)
• Hugging Face Transformers or local LLM via HTTP
• Standard Python libraries + custom utils modules