# opendatalab/provergen architecture

> Architecture diagram of the GitHub repository opendatalab/provergen, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opendatalab/provergen
- Repository: https://github.com/opendatalab/provergen

## Overview

##### Project type and purpose
- Project type: A command-line, data-generation and evaluation framework (open-source tool) for creating logical reasoning problems (ProverGen).
- Purpose:
- Generate first-order logic (FOL) “skeletons” using a symbolic prover (Prover9)
- Translate skeletons into natural language via large language models (LLMs)
- Produce final FOL reasoning problems (with augmentations)
- Evaluate LLM performance against various datasets

##### File structure
- Top-level scripts:
- logic_skeleton_generator.py → invokes Prover9 to produce logic skeletons
- logic_skeleton_translator.py → uses LLMs to translate skeletons to nat-lang
- fol_problem_generator.py → generates FOL problems with distractions/augmentations
- evaluation.py → runs LLMs on generated datasets
- result/metrics.py → computes metrics on evaluation outputs
- Data directories:
- data/ → rule pools, name lists, example translations
- logic_data/ → skeleton outputs and few-shot examples for each benchmark
- training_data/ → prebuilt datasets (folio, proofwriter…)
- Utilities under utils/:
- llms_interface.py, prover9_interface.py → abstract external services
- augmentor.py → data augmentation
- logic_generator/ & logic_translator/ → internal pipeline components
- Configuration: requirements.txt, .gitignore

##### README for additional insights
- Describes CLI parameters for each stage (num, mode, start/end, noise flags)
- Relies on external services: Prover9 binary + LLM API (HuggingFace or local server)

a. Main components:
1. Data Storage
- Raw resources (data/, training_data/)
- Intermediate output (logic_data/, translated_data/)
- Final problem sets + evaluation results (outputs/, result/)
2. Logic Skeleton Generator
- Script: logic_skeleton_generator.py
- Calls Prover9 via utils/prover9_interface.py
3. Logic Skeleton Translator
- Script: logic_skeleton_translator.py
- Calls LLMs via utils/llms_interface.py
4. FOL Problem Generator
- Script: fol_problem_generator.py
- Uses augmentor.py for noise and paraphrase
5. Evaluation Module
- Script: evaluation.py
- Metrics computed by result/metrics.py
6. External Services
- Prover9 binary (symbolic prover)
- LLM service (Hugging Face or local)
b. Component interactions (data flow):
Data resources → Logic Skeleton Generator → logic_data/
logic_data/ + LLM Interface → Logic Skeleton Translator → translated_data/
translated_data/ + augmentor → FOL Problem Generator → final_data/
final_data/ + LLM Interface → Evaluation → result/ → metrics computation
c. Architectural patterns / principles:
- Modular pipeline – each stage is an independent, reusable script
- Adapter pattern – utils/llms_interface.py and prover9_interface.py abstract external services
- Data-driven – JSON files passed between stages
d. Key technologies / frameworks:
- Python 3, JSON, Conda environment
- Prover9 (symbolic prover)
- Hugging Face Transformers or local LLM via HTTP
- Standard Python libraries + custom utils modules

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Storage
    raw["Raw Resources\n(data/)"]:::storage
    train["Prebuilt Datasets\n(training_data/)"]:::storage
    skeleton["Intermediate Output\n(logic_data/)"]:::storage
    translated["Translated Data\n(translated_data/)"]:::storage
    final["Final Problems\n(final_data/)"]:::storage
    result_store["Evaluation Outputs\n(result/)"]:::storage
    metrics["Metrics Computation\n(result/metrics.py)"]:::storage

    %% Processing Modules
    logic_gen["Logic Skeleton Generator\n(logic_skeleton_generator.py)"]:::process
    translator["Logic Skeleton Translator\n(logic_skeleton_translator.py)"]:::process
    problem_gen["FOL Problem Generator\n(fol_problem_generator.py)"]:::process
    evaluation["Evaluation Module\n(evaluation.py)"]:::process

    %% Adapters & Helpers
    prov_adapter["Prover9 Adapter\n(utils/prover9_interface.py)"]:::adapter
    logic_helpers["Logic Gen Helpers\n(utils/logic_generator/)"]:::helper
    llm_adapter["LLM Adapter\n(utils/llms_interface.py)"]:::adapter
    translation_helpers["Logic Translation Helpers\n(utils/logic_translator/)"]:::helper
    augmentor["Augmentor\n(utils/augmentor.py)"]:::helper

    %% External Services
    external_prover["Prover9 Binary"]:::external
    external_llm["LLM Service"]:::external

    %% Data Flow
    raw -->|".json"| logic_gen
    train -->|".json"| logic_gen
    logic_gen --> skeleton
    skeleton -->|".json"| translator
    translated -->|".json"| problem_gen
    problem_gen --> final
    final -->|".json"| evaluation
    evaluation --> result_store
    result_store --> metrics

    %% Adapter & Helpers Flow
    logic_gen --> prov_adapter
    prov_adapter --> external_prover
    prov_adapter --> logic_helpers
    translator --> llm_adapter
    llm_adapter --> external_llm
    translator --> translation_helpers
    problem_gen --> augmentor
    evaluation --> llm_adapter

    %% Click Events
    click raw "https://github.com/opendatalab/provergen/tree/main/data/"
    click train "https://github.com/opendatalab/provergen/tree/main/training_data/"
    click skeleton "https://github.com/opendatalab/provergen/tree/main/logic_data/"
    click result_store "https://github.com/opendatalab/provergen/tree/main/result/"
    click metrics "https://github.com/opendatalab/provergen/blob/main/result/metrics.py"
    click logic_gen "https://github.com/opendatalab/provergen/blob/main/logic_skeleton_generator.py"
    click prov_adapter "https://github.com/opendatalab/provergen/blob/main/utils/prover9_interface.py"
    click logic_helpers "https://github.com/opendatalab/provergen/tree/main/utils/logic_generator/"
    click translator "https://github.com/opendatalab/provergen/blob/main/logic_skeleton_translator.py"
    click llm_adapter "https://github.com/opendatalab/provergen/blob/main/utils/llms_interface.py"
    click translation_helpers "https://github.com/opendatalab/provergen/tree/main/utils/logic_translator/"
    click problem_gen "https://github.com/opendatalab/provergen/blob/main/fol_problem_generator.py"
    click augmentor "https://github.com/opendatalab/provergen/blob/main/utils/augmentor.py"
    click evaluation "https://github.com/opendatalab/provergen/blob/main/evaluation.py"
    click metrics "https://github.com/opendatalab/provergen/blob/main/result/metrics.py"

    %% Styling
    classDef storage fill:#cce5ff,stroke:#004085,color:#004085,stroke-width:1px;
    classDef process fill:#d4edda,stroke:#155724,color:#155724,stroke-width:1px;
    classDef adapter fill:#ffeeba,stroke:#856404,color:#856404,stroke-width:1px;
    classDef helper fill:#e2e3e5,stroke:#6c757d,color:#6c757d,stroke-width:1px;
    classDef external fill:#f8d7da,stroke:#721c24,color:#721c24,stroke-width:1px;
```
