# iaar-shanghai/xverify architecture

> Architecture diagram of the GitHub repository iaar-shanghai/xverify, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/iaar-shanghai/xverify
- Repository: https://github.com/iaar-shanghai/xverify

## Overview

##### Project type and purpose
- xVerify is an open-source Python evaluation tool (not a full-stack app) that wraps a fine-tuned LLM to verify LLM-generated answers to objective questions.
- It loads question/answer pairs, constructs verification prompts, invokes an LLM (locally or via API), and applies equivalence logic to judge correctness.

##### File structure
- Top-level:
- README.md, LICENSE.txt, requirements.txt, demo.ipynb (usage notebook)
- assets (figures for the paper)
- src/xVerify: core package
- dataset_loader.py – reads JSON data
- model.py – Model class abstracts LLM invocation (local via vLLM or remote via API)
- prompts/ – prompt-template classes (base_template.py, judge_prompt.py)
- eval.py – Evaluator class orchestrates the end-to-end evaluation
- examples/ – sample JSON for batch evaluation

##### README for additional insights
- Describes:
- Supported question types (math, multiple-choice, classification, short answer)
- Model variants (0.5B–32B) and deployment via HuggingFace/vLLM or OpenAI API
- Setup steps (conda env, pip install, model download)
- Demo code for single and batch evaluation

1. Data Layer
– Input dataset (eval_examples.json or user JSON)
– dataset_loader.py
2. Prompt Builder
– base_template.py
– judge_prompt.py
3. Model Layer
– Model class (src/xVerify/model.py)
– Inference backend:
- vLLM server (local)
- OpenAI API (remote)
4. Evaluation Layer
– Evaluator class (src/xVerify/eval.py)
– Equivalence logic (handled by LLM + post-processing)
5. Presentation Layer
– demo.ipynb / CLI (user entry points)
1. User supplies dataset → dataset_loader → Python objects
2. Evaluator requests Model → Prompt Builder formats prompt → send to LLM backend
3. LLM backend returns llm_output → Evaluator compares to correct_answer → judgment
4. Results returned to notebook or saved to disk
c. Architectural patterns & principles:
- Layered architecture (Data → Prompt → Model → Evaluation → Output)
- Separation of concerns: loader, prompt templates, inference, evaluation logic
- Extensibility: new prompt templates or backends can be plugged in
d. Relevant technologies/frameworks:
- Python 3.10, Conda for environment
- HuggingFace transformers / vLLM for local inference
- OpenAI API support
- JSON for data interchange
- Jupyter notebook for demonstration

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Data Layer"
        Data_Loader["dataset_loader.py"]:::data
        Examples[(eval_examples.json)]:::data
    end

    subgraph "Prompt Builder"
        Base_Template["base_template.py"]:::process
        Judge_Prompt["judge_prompt.py"]:::process
    end

    subgraph "Model Layer"
        Model["model.py"]:::process
    end

    subgraph "External Inference Services"
        vLLM["vLLM Server"]:::external
        OpenAI["OpenAI API"]:::external
    end

    subgraph "Evaluation Layer"
        Evaluator["eval.py"]:::process
        Equivalence["Equivalence Logic"]:::process
    end

    subgraph "Presentation Layer"
        Notebook["demo.ipynb"]:::presentation
    end

    Examples -->|"JSON data"| Data_Loader
    Data_Loader -->|"Python objects"| Evaluator

    Evaluator -->|"format using templates"| Base_Template
    Evaluator -->|"format using templates"| Judge_Prompt
    Base_Template -->|"template"| Model
    Judge_Prompt -->|"template"| Model

    Model -->|"inference request"| vLLM
    Model -->|"inference request"| OpenAI
    vLLM -->|"llm_response"| Evaluator
    OpenAI -->|"llm_response"| Evaluator

    Evaluator -->|"apply equivalence logic"| Equivalence
    Equivalence -->|"judgment result"| Evaluator

    Evaluator -->|"results"| Notebook

    click Data_Loader "https://github.com/iaar-shanghai/xverify/blob/main/src/xVerify/dataset_loader.py"
    click Examples "https://github.com/iaar-shanghai/xverify/blob/main/src/xVerify/examples/eval_examples.json"
    click Base_Template "https://github.com/iaar-shanghai/xverify/blob/main/src/xVerify/prompts/base_template.py"
    click Judge_Prompt "https://github.com/iaar-shanghai/xverify/blob/main/src/xVerify/prompts/judge_prompt.py"
    click Model "https://github.com/iaar-shanghai/xverify/blob/main/src/xVerify/model.py"
    click Evaluator "https://github.com/iaar-shanghai/xverify/blob/main/src/xVerify/eval.py"
    click Notebook "https://github.com/iaar-shanghai/xverify/blob/main/demo.ipynb"

    classDef data fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef process fill:#90EE90,stroke:#333,stroke-width:1px
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px
    classDef presentation fill:#D3D3D3,stroke:#333,stroke-width:1px
```
