How iaar-shanghai/xverify works
Project type and purpose
• xVerify is an open-source Python evaluation tool (not a full-stack app) that wraps a fine-tuned LLM to verify LLM-generated answers to objective questions.
• It loads question/answer pairs, constructs verification prompts, invokes an LLM (locally or via API), and applies equivalence logic to judge correctness.
File structure
• Top-level:
• README.md, LICENSE.txt, requirements.txt, demo.ipynb (usage notebook)
• assets (figures for the paper)
• src/xVerify: core package
• dataset_loader.py – reads JSON data
• model.py – Model class abstracts LLM invocation (local via vLLM or remote via API)
• prompts/ – prompt-template classes (base_template.py, judge_prompt.py)
• eval.py – Evaluator class orchestrates the end-to-end evaluation
• examples/ – sample JSON for batch evaluation
README for additional insights
• Describes:
• Supported question types (math, multiple-choice, classification, short answer)
• Model variants (0.5B–32B) and deployment via HuggingFace/vLLM or OpenAI API
• Setup steps (conda env, pip install, model download)
• Demo code for single and batch evaluation
1. Data Layer
– Input dataset (eval_examples.json or user JSON)
– dataset_loader.py
2. Prompt Builder
– base_template.py
– judge_prompt.py
3. Model Layer
– Model class (src/xVerify/model.py)
– Inference backend:
• vLLM server (local)
• OpenAI API (remote)
4. Evaluation Layer
– Evaluator class (src/xVerify/eval.py)
– Equivalence logic (handled by LLM + post-processing)
5. Presentation Layer
– demo.ipynb / CLI (user entry points)
1. User supplies dataset → dataset_loader → Python objects
2. Evaluator requests Model → Prompt Builder formats prompt → send to LLM backend
3. LLM backend returns llm_output → Evaluator compares to correct_answer → judgment
4. Results returned to notebook or saved to disk
c. Architectural patterns & principles:
• Layered architecture (Data → Prompt → Model → Evaluation → Output)
• Separation of concerns: loader, prompt templates, inference, evaluation logic
• Extensibility: new prompt templates or backends can be plugged in
d. Relevant technologies/frameworks:
• Python 3.10, Conda for environment
• HuggingFace transformers / vLLM for local inference
• OpenAI API support
• JSON for data interchange
• Jupyter notebook for demonstration