How jlko/semantic_uncertainty works
Project type and purpose
• This is a research‐oriented machine‐learning pipeline for “Detecting Hallucinations in Large Language Models Using Semantic Entropy.”
• It is not a web or full‐stack application but an experimental framework: you feed in datasets, generate model responses, compute uncertainty measures, and analyze results.
File structure
• Top level:
• notebooks/ – Jupyter notebooks for evaluation and plotting.
• semantic_uncertainty/ – core Python package.
• environment*.yaml – conda environment specs.
• README.md, LICENSE – documentation.
• Inside semantic_uncertainty/:
• generate_answers.py – script to sample from LLMs.
• compute_uncertainty_measures.py – script to compute metrics.
• uncertainty/ – subpackage:
• data/ – dataset loaders and utilities.
• models/ – base classes and HuggingFace wrappers.
• uncertainty_measures/ – implementations of p_ik, p_true, semantic entropy.
• utils/ – evaluation helpers, API clients (OpenAI, WandB), etc.
README for additional insights
• The pipeline runs in three sequential steps (generate → compute → analyze).
• Dependencies: Python 3.11, PyTorch 2.1, HuggingFace Transformers, OpenAI API, WandB.
• Data is fetched from Hugging Face or manually for BioASQ.
• GPU‐accelerated inference essential.