How alibaba/eigenscore works
Project Type and Purpose
• This is an open‐source research codebase for “INSIDE: LLMs’ Internal States Retain the Power of Hallucination Detection,” an ML experiment and evaluation toolkit.
• It provides data loaders, evaluation scripts, model wrappers (including OpenAI and custom NLI models), metric calculators, plotting utilities, and a pipeline to generate and assess hallucination detection performance.
Top‐Level Components (from file tree)
a. Data Layer
• data/datasets/… (raw JSON datasets, fig1.pdf)
b. Data Evaluation Layer
• dataeval/ (load.py, coqa.py, SQuAD.py, triviaqa.py, load_worker.py)
c. Model Abstraction Layer
• models/_load_model.py
• models/nli.py
• models/openai_models.py
d. Functional Utilities
• func/evalFunc.py (core evaluation routines)
• func/metric.py (metric definitions)
• func/plot.py (visualization)
e. Orchestration / Pipeline
• pipeline/generate.py (main logic to tie together data, models, and evaluation)
• run.sh (shell entrypoint)
f. General Utilities
• utils/parallel.py (multi‐process support)
Key Interactions and Data Flows
• run.sh → invokes pipeline/generate.py
• pipeline/generate.py
→ loads dataset(s) via dataeval/load.py (and dataset‐specific modules)
→ loads or initializes model(s) via models/_load_model.py → models/openai_models.py or models/nli.py
→ for each data instance: calls model to produce outputs
→ passes predictions+references to func/evalFunc.py & func/metric.py
→ optionally calls func/plot.py to generate figures
→ uses utils/parallel.py to parallelize across CPU/GPU
• dataeval/load_worker.py: supports dataeval’s parallel loading
Architectural Patterns and Principles
• Modular pipeline with clear separation of concerns: Data I/O → Modeling → Evaluation → Visualization
• Abstraction of external services (OpenAI API) behind model wrapper interface
• Pluggable dataset handlers for CoQA, SQuAD, TriviaQA, etc.
• Utility‐based parallelism to scale experiments