# gaurangsriramanan/llm_check_hallucination_detection architecture

> Architecture diagram of the GitHub repository gaurangsriramanan/llm_check_hallucination_detection, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gaurangsriramanan/llm_check_hallucination_detection
- Repository: https://github.com/gaurangsriramanan/llm_check_hallucination_detection

## Overview

##### Project type and purpose
- This is a research‐oriented codebase (“LLM-Check”) for detecting hallucinations in LLM responses. It runs on diverse datasets (FAVA, SelfCheck, RAGTruth) and applies multiple scoring techniques (eigen-analysis of attention/hidden states, token uncertainty) to flag hallucinated tokens.

##### File structure
Top-level directories and files:
- run_detection_combined.py & run.sh: orchestration scripts
- common_utils.py, utils_fava.py/annotated, utils_ragtruth.py, utils_selfcheck.py, utils_selfcheck.py: dataset loaders + dataset-specific iterators
- data/: Persistent storage for raw datasets and computed scores (.pkl)
- logs/: stdout/stderr logs from model runs
- plots/: output visualizations
- check_scores_fava_annotated.ipynb: notebook for score analysis/plots
- environment.yml: conda environment specification
- Hallucination_Detection_NeurIPS_24.pdf & README.md: paper + project description

##### Additional insights from README
- Core steps: load dataset → feed examples to Llama-2-7B Chat → extract hidden states, attention kernels, logits → compute multiple scores → save to disk → analyze via notebook.
- External dependency: HuggingFace Transformers for model inference.
- Output directory (/data) holds pickle files of scores for downstream analysis.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        direction TB
        DataRaw["data/ (Raw Datasets)"]:::data
        UtilsFAVA["utils_fava.py"]:::compute
        UtilsFAVAAnnot["utils_fava_annotated.py"]:::compute
        UtilsRAG["utils_ragtruth.py"]:::compute
        UtilsSelf["utils_selfcheck.py"]:::compute
    end

    %% Orchestration & Control
    subgraph "Orchestration & Control"
        direction TB
        RunSH["run.sh"]:::orchestration
        RunDetect["run_detection_combined.py"]:::orchestration
    end

    %% Model Inference Module
    subgraph "Model Inference Module"
        direction TB
        CommonUtils["common_utils.py"]:::compute
        HuggingFace["Llama-2 Chat Model"]:::external
    end

    %% Feature Extraction & Scoring
    subgraph "Feature Extraction & Scoring"
        direction TB
        EigenModule["Eigenvalue Scoring\n(attention & hidden)"]:::compute
        UncertaintyModule["Token Uncertainty\n(perplexity & entropy)"]:::compute
    end

    %% Persistence & Logging
    subgraph "Persistence & Logging"
        direction TB
        DataStore["data/ (Scores .pkl)"]:::data
        Logs["logs/"]:::data
    end

    %% Post-Processing & Visualization
    subgraph "Post-Processing & Visualization"
        direction TB
        Notebook["check_scores_fava_annotated.ipynb"]:::analysis
        PlotsDir["plots/"]:::analysis
    end

    %% Environment & Documentation
    subgraph "Environment & Documentation"
        direction TB
        EnvFile["environment.yml"]:::docs
        Readme["README.md"]:::docs
        Paper["Hallucination_Detection_NeurIPS_24.pdf"]:::docs
    end

    %% Connections
    DataRaw -->|loads via| UtilsFAVA
    DataRaw -->|loads via| UtilsFAVAAnnot
    DataRaw -->|loads via| UtilsRAG
    DataRaw -->|loads via| UtilsSelf

    RunSH -->|calls| RunDetect
    RunDetect -->|uses loaders| UtilsFAVA
    RunDetect -->|uses loaders| UtilsRAG
    RunDetect -->|uses loaders| UtilsSelf
    RunDetect -->|invokes| CommonUtils

    CommonUtils -->|queries| HuggingFace
    CommonUtils -->|returns hidden & attention| EigenModule
    CommonUtils -->|returns logits| UncertaintyModule

    EigenModule -->|writes scores| DataStore
    UncertaintyModule -->|writes scores| DataStore

    RunDetect -->|writes logs| Logs
    EigenModule -->|writes logs| Logs
    UncertaintyModule -->|writes logs| Logs

    Notebook -->|reads scores| DataStore
    Notebook -->|generates plots| PlotsDir

    %% Click Events
    click DataRaw "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/tree/main/data/"
    click UtilsFAVA "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/blob/main/utils_fava.py"
    click UtilsFAVAAnnot "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/blob/main/utils_fava_annotated.py"
    click UtilsRAG "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/blob/main/utils_ragtruth.py"
    click UtilsSelf "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/blob/main/utils_selfcheck.py"
    click RunSH "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/blob/main/run.sh"
    click RunDetect "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/blob/main/run_detection_combined.py"
    click CommonUtils "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/blob/main/common_utils.py"
    click DataStore "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/tree/main/data/"
    click Logs "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/tree/main/logs/"
    click Notebook "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/blob/main/check_scores_fava_annotated.ipynb"
    click PlotsDir "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/tree/main/plots/"
    click EnvFile "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/blob/main/environment.yml"
    click Readme "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/blob/main/README.md"
    click Paper "https://github.com/gaurangsriramanan/llm_check_hallucination_detection/blob/main/Hallucination_Detection_NeurIPS_24.pdf"

    %% Styles
    classDef data fill:#ccffcc,stroke:#2d572c,stroke-width:1px;
    classDef compute fill:#cce5ff,stroke:#1a237e,stroke-width:1px;
    classDef orchestration fill:#ffe0b2,stroke:#e65100,stroke-width:1px;
    classDef external fill:#e1bee7,stroke:#4a148c,stroke-width:1px;
    classDef analysis fill:#f0f4c3,stroke:#827717,stroke-width:1px;
    classDef docs fill:#eeeeee,stroke:#616161,stroke-width:1px;
```
