How gaurangsriramanan/llm_check_hallucination_detection works
Project type and purpose
• This is a research‐oriented codebase (“LLM-Check”) for detecting hallucinations in LLM responses. It runs on diverse datasets (FAVA, SelfCheck, RAGTruth) and applies multiple scoring techniques (eigen-analysis of attention/hidden states, token uncertainty) to flag hallucinated tokens.
File structure
Top-level directories and files:
• run_detection_combined.py & run.sh: orchestration scripts
• common_utils.py, utils_fava.py/annotated, utils_ragtruth.py, utils_selfcheck.py, utils_selfcheck.py: dataset loaders + dataset-specific iterators
• data/: Persistent storage for raw datasets and computed scores (.pkl)
• logs/: stdout/stderr logs from model runs
• plots/: output visualizations
• check_scores_fava_annotated.ipynb: notebook for score analysis/plots
• environment.yml: conda environment specification
• Hallucination_Detection_NeurIPS_24.pdf & README.md: paper + project description
Additional insights from README
• Core steps: load dataset → feed examples to Llama-2-7B Chat → extract hidden states, attention kernels, logits → compute multiple scores → save to disk → analyze via notebook.
• External dependency: HuggingFace Transformers for model inference.
• Output directory (/data) holds pickle files of scores for downstream analysis.