How nlpyang/geval works
Project type and purpose
• This is a research-oriented evaluation pipeline (not a full-stack web app).
• Purpose: take the SummEval dataset, feed examples plus evaluation prompts into GPT-4, collect its judgments, then run meta-analysis on those judgments.
File structure
• Top-level scripts:
– gpt4_eval.py: drives the main evaluation loop.
– meta_eval_summeval.py: runs post-processing/meta-analysis on saved GPT-4 outputs.
• Data directories:
– data/summeval.json: input dataset.
– prompts/summeval/*.txt: a set of detailed prompt templates, one per dimension (fluency, coherence, consistency, relevance).
• Output directories:
– results/*.json: GPT-4’s per-dimension judgments.
• Config/dependency cues:
– You call GPT-4 via an API key (passed at CLI).
– Scripts read/write JSON, use prompt templates.
README for additional insights
• Two-stage CLI usage:
a) Evaluate a dimension: gpt4_eval.py --prompt <template> --summeval_fp <data> --save_fp <results> --key <API-key>
b) Meta-evaluate: meta_eval_summeval.py --input_fp <results> --dimension <name>
1) Dataset Loader (reads data/summeval.json)
2) Prompt Manager (loads one of the prompts/*.txt)
3) GPT-4 API Client (sends prompt + example, receives score)
4) Results Store (writes JSON to results/)
5) Meta-Evaluation Processor (loads results JSON, computes stats)
6) CLI Wrapper (parses arguments, orchestrates calls)
– Dataset Loader → CLI Wrapper
– Prompt Manager → CLI Wrapper
– CLI Wrapper → GPT-4 API Client → GPT-4 Service → returns to Client → CLI Wrapper
– CLI Wrapper → Results Store
– Meta-Evaluation Processor reads from Results Store → outputs summary to stdout or file
c. Architectural patterns / design principles:
– Pipeline pattern: sequential stages (load → prompt → evaluate → store → analyze).
– Separation of concerns: data I/O, prompting logic, API interaction, post-processing.
d. Key technologies & libraries:
– Python 3.x scripts
– OpenAI GPT-4 API
– JSON for data interchange
– Filesystem for storing prompts and results
Detailed coverage is key
– Show all folders you see: data/, prompts/, results/.
– Represent the filesystem (for prompts and JSON) as a “Data Store” cluster.