# alibaba/eigenscore architecture

> Architecture diagram of the GitHub repository alibaba/eigenscore, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alibaba/eigenscore
- Repository: https://github.com/alibaba/eigenscore

## Overview

##### Project Type and Purpose
- This is an open‐source research codebase for “INSIDE: LLMs’ Internal States Retain the Power of Hallucination Detection,” an ML experiment and evaluation toolkit.
- It provides data loaders, evaluation scripts, model wrappers (including OpenAI and custom NLI models), metric calculators, plotting utilities, and a pipeline to generate and assess hallucination detection performance.

##### Top‐Level Components (from file tree)
a. Data Layer
- data/datasets/… (raw JSON datasets, fig1.pdf)
b. Data Evaluation Layer
- dataeval/ (load.py, coqa.py, SQuAD.py, triviaqa.py, load_worker.py)
c. Model Abstraction Layer
- models/_load_model.py
- models/nli.py
- models/openai_models.py
d. Functional Utilities
- func/evalFunc.py (core evaluation routines)
- func/metric.py (metric definitions)
- func/plot.py (visualization)
e. Orchestration / Pipeline
- pipeline/generate.py (main logic to tie together data, models, and evaluation)
- run.sh (shell entrypoint)
f. General Utilities
- utils/parallel.py (multi‐process support)

##### Key Interactions and Data Flows
- run.sh → invokes pipeline/generate.py
- pipeline/generate.py
→ loads dataset(s) via dataeval/load.py (and dataset‐specific modules)
→ loads or initializes model(s) via models/_load_model.py → models/openai_models.py or models/nli.py
→ for each data instance: calls model to produce outputs
→ passes predictions+references to func/evalFunc.py & func/metric.py
→ optionally calls func/plot.py to generate figures
→ uses utils/parallel.py to parallelize across CPU/GPU
- dataeval/load_worker.py: supports dataeval’s parallel loading

##### Architectural Patterns and Principles
- Modular pipeline with clear separation of concerns: Data I/O → Modeling → Evaluation → Visualization
- Abstraction of external services (OpenAI API) behind model wrapper interface
- Pluggable dataset handlers for CoQA, SQuAD, TriviaQA, etc.
- Utility‐based parallelism to scale experiments

## Diagram (Mermaid)

```mermaid
flowchart TD
    run_sh["run.sh (Shell Entrypoint)"]:::entry
    subgraph "Pipeline Orchestrator"
        gen["generate.py"]:::processing
    end
    subgraph "Data Sources"
        ds["Datasets (JSON, fig1.pdf)"]:::data
    end
    subgraph "DataEval Module"
        dl["load.py"]:::processing
        coqa["coqa.py"]:::processing
        squad["SQuAD.py"]:::processing
        trivia["triviaqa.py"]:::processing
        nq["nq_open.py"]:::processing
        lw["load_worker.py"]:::processing
    end
    subgraph "Model Layer"
        load_mod["_load_model.py"]:::processing
        nli["nli.py"]:::processing
        openai["openai_models.py"]:::external
    end
    subgraph "Evaluation & Metrics"
        evalf["evalFunc.py"]:::processing
        metric["metric.py"]:::processing
    end
    vis["plot.py"]:::processing
    subgraph "Utilities"
        par["parallel.py"]:::utility
    end
    cloudOpenAI("Cloud: OpenAI API"):::external

    run_sh -->|invokes| gen
    ds -->|load JSON| dl
    coqa --> dl
    squad --> dl
    trivia --> dl
    nq --> dl
    lw --> dl
    dl -->|provides data| gen
    gen -->|initialize models| load_mod
    load_mod -->|instantiate NLI| nli
    load_mod -->|instantiate OpenAI| openai
    openai -->|API calls| cloudOpenAI
    gen -->|generate outputs| nli
    gen -->|generate outputs| openai
    gen -->|compute metrics| metric
    gen -->|evaluate predictions| evalf
    gen -->|plot figures| vis
    gen -->|spawn workers| par

    click run_sh "https://github.com/alibaba/eigenscore/blob/main/run.sh"
    click gen "https://github.com/alibaba/eigenscore/blob/main/pipeline/generate.py"
    click ds "https://github.com/alibaba/eigenscore/tree/main/data/datasets/"
    click dl "https://github.com/alibaba/eigenscore/blob/main/dataeval/load.py"
    click coqa "https://github.com/alibaba/eigenscore/blob/main/dataeval/coqa.py"
    click squad "https://github.com/alibaba/eigenscore/blob/main/dataeval/SQuAD.py"
    click trivia "https://github.com/alibaba/eigenscore/blob/main/dataeval/triviaqa.py"
    click nq "https://github.com/alibaba/eigenscore/blob/main/dataeval/nq_open.py"
    click lw "https://github.com/alibaba/eigenscore/blob/main/dataeval/load_worker.py"
    click load_mod "https://github.com/alibaba/eigenscore/blob/main/models/_load_model.py"
    click nli "https://github.com/alibaba/eigenscore/blob/main/models/nli.py"
    click openai "https://github.com/alibaba/eigenscore/blob/main/models/openai_models.py"
    click evalf "https://github.com/alibaba/eigenscore/blob/main/func/evalFunc.py"
    click metric "https://github.com/alibaba/eigenscore/blob/main/func/metric.py"
    click vis "https://github.com/alibaba/eigenscore/blob/main/func/plot.py"
    click par "https://github.com/alibaba/eigenscore/blob/main/utils/parallel.py"

    classDef data fill:#cce5ff,stroke:#000;
    classDef processing fill:#d4f4dd,stroke:#000;
    classDef external fill:#ffe5b4,stroke:#000;
    classDef utility fill:#e0e0e0,stroke:#000;
    classDef entry fill:#f0f0f0,stroke:#000;
```
