How d2i-ai/eigenscore works
Project type and purpose
• This is a research‐code repository implementing the INSIDE hallucination‐detection method for large language models (LLMs).
• It provides data processing, model‐interface layers, evaluation metrics, and plotting utilities to run experiments on question‐answer datasets (SQuAD, CoQA, NaturalQuestions, etc.).
File structure
• Top‐level scripts and configs:
• run.sh – orchestrates end‐to‐end execution.
• _settings.py – global configuration (paths, hyperparameters).
• Data storage:
• data/datasets/*.json, fig1.pdf – raw datasets and reference figures.
• Data loading & evaluation:
• dataeval/ – modules for loading each dataset, abstraction for evaluation tasks.
• Core functions:
• func/ – evaluation functions (evalFunc.py), metric computation (metric.py), plotting (plot.py).
• Modeling layer:
• models/ – model‐loading logic (_load_model.py), interfaces to NLI and OpenAI models (nli.py, openai_models.py).
• Pipeline orchestration:
• pipeline/generate.py – main script to generate LLM outputs and internal states for eigenscore.
• Utilities:
• utils/parallel.py – helper for multiprocessing and parallel data processing.
README for additional insights
• The README confirms it is a code release for a paper at ICLR 2024 on hallucination detection.
Elements
• Clear labels: “Dataset Storage,” “Data Loader,” “Model Interface,” “Inference Engine,” “Evaluation,” “Plotting,” “Utilities,” “Orchestrator.”
• Datasets → Data Loader
• Data Loader → Inference Engine
• Inference Engine → Evaluation
• Evaluation → Plotting
• Orchestrator → all modules
• Rectangles (green) for processing modules
• Parallelograms (orange) for scripts/orchestrator
Detailed guidance
• Run.sh/_settings.py at the top or side to show centralized orchestration.
• Modules from left to right or top to bottom in data‐flow order.