# nlpyang/geval architecture

> Architecture diagram of the GitHub repository nlpyang/geval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nlpyang/geval
- Repository: https://github.com/nlpyang/geval

## Overview

##### Project type and purpose
- This is a research-oriented evaluation pipeline (not a full-stack web app).
- Purpose: take the SummEval dataset, feed examples plus evaluation prompts into GPT-4, collect its judgments, then run meta-analysis on those judgments.

##### File structure
- Top-level scripts:
– gpt4_eval.py: drives the main evaluation loop.
– meta_eval_summeval.py: runs post-processing/meta-analysis on saved GPT-4 outputs.
- Data directories:
– data/summeval.json: input dataset.
– prompts/summeval/*.txt: a set of detailed prompt templates, one per dimension (fluency, coherence, consistency, relevance).
- Output directories:
– results/*.json: GPT-4’s per-dimension judgments.
- Config/dependency cues:
– You call GPT-4 via an API key (passed at CLI).
– Scripts read/write JSON, use prompt templates.

##### README for additional insights
- Two-stage CLI usage:
a) Evaluate a dimension: `gpt4_eval.py --prompt <template> --summeval_fp <data> --save_fp <results> --key <API-key>`
b) Meta-evaluate: `meta_eval_summeval.py --input_fp <results> --dimension <name>`

1) Dataset Loader (reads data/summeval.json)
2) Prompt Manager (loads one of the prompts/*.txt)
3) GPT-4 API Client (sends prompt + example, receives score)
4) Results Store (writes JSON to results/)
5) Meta-Evaluation Processor (loads results JSON, computes stats)
6) CLI Wrapper (parses arguments, orchestrates calls)
– Dataset Loader → CLI Wrapper
– Prompt Manager → CLI Wrapper
– CLI Wrapper → GPT-4 API Client → GPT-4 Service → returns to Client → CLI Wrapper
– CLI Wrapper → Results Store
– Meta-Evaluation Processor reads from Results Store → outputs summary to stdout or file
c. Architectural patterns / design principles:
– Pipeline pattern: sequential stages (load → prompt → evaluate → store → analyze).
– Separation of concerns: data I/O, prompting logic, API interaction, post-processing.
d. Key technologies & libraries:
– Python 3.x scripts
– OpenAI GPT-4 API
– JSON for data interchange
– Filesystem for storing prompts and results

##### Detailed coverage is key
– Show all folders you see: data/, prompts/, results/.
– Represent the filesystem (for prompts and JSON) as a “Data Store” cluster.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Store"
        DL["Dataset Loader\n(data/summeval.json)"]:::data
        PM["Prompt Manager\n(prompts/summeval/*.txt)"]:::data
        RS["Results Store\n(results/*.json)"]:::data
    end

    subgraph "Orchestration"
        CLI1["CLI Wrapper\n(gpt4_eval.py)"]:::logic
        GPTClient["GPT-4 API Client\n(gpt4_eval.py)"]:::logic
        CLI2["CLI Wrapper\n(meta_eval_summeval.py)"]:::logic
        MetaProc["Meta-Evaluation Processor\n(meta_eval_summeval.py)"]:::logic
    end

    ES["GPT-4 Service"]:::external

    DL --> CLI1
    PM --> CLI1
    CLI1 --> GPTClient
    GPTClient --> ES
    ES --> GPTClient
    GPTClient --> CLI1
    CLI1 --> RS

    RS --> CLI2
    CLI2 --> MetaProc

    click DL "https://github.com/nlpyang/geval/blob/main/data/summeval.json"
    click PM "https://github.com/nlpyang/geval/tree/main/prompts/summeval/"
    click GPTClient "https://github.com/nlpyang/geval/blob/main/gpt4_eval.py"
    click CLI1 "https://github.com/nlpyang/geval/blob/main/gpt4_eval.py"
    click CLI2 "https://github.com/nlpyang/geval/blob/main/meta_eval_summeval.py"
    click MetaProc "https://github.com/nlpyang/geval/blob/main/meta_eval_summeval.py"
    click RS "https://github.com/nlpyang/geval/tree/main/results/"

    classDef data fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef logic fill:#90EE90,stroke:#000,stroke-width:1px
    classDef external fill:#FFA500,stroke:#000,stroke-width:1px
```
