# jlko/semantic_uncertainty architecture

> Architecture diagram of the GitHub repository jlko/semantic_uncertainty, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jlko/semantic_uncertainty
- Repository: https://github.com/jlko/semantic_uncertainty

## Overview

##### Project type and purpose
- This is a research‐oriented machine‐learning pipeline for “Detecting Hallucinations in Large Language Models Using Semantic Entropy.”
- It is not a web or full‐stack application but an experimental framework: you feed in datasets, generate model responses, compute uncertainty measures, and analyze results.

##### File structure
- Top level:
- notebooks/ – Jupyter notebooks for evaluation and plotting.
- semantic_uncertainty/ – core Python package.
- environment*.yaml – conda environment specs.
- README.md, LICENSE – documentation.
- Inside semantic_uncertainty/:
- generate_answers.py – script to sample from LLMs.
- compute_uncertainty_measures.py – script to compute metrics.
- uncertainty/ – subpackage:
- data/ – dataset loaders and utilities.
- models/ – base classes and HuggingFace wrappers.
- uncertainty_measures/ – implementations of p_ik, p_true, semantic entropy.
- utils/ – evaluation helpers, API clients (OpenAI, WandB), etc.

##### README for additional insights
- The pipeline runs in three sequential steps (generate → compute → analyze).
- Dependencies: Python 3.11, PyTorch 2.1, HuggingFace Transformers, OpenAI API, WandB.
- Data is fetched from Hugging Face or manually for BioASQ.
- GPU‐accelerated inference essential.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "External Services"
        HuggingFace["HuggingFace Hub"]:::service
        OpenAI["OpenAI API"]:::service
        WandB["Weights & Biases"]:::service
    end

    DataLayer["Data Layer\n(dataset loaders)"]:::module
    click DataLayer "https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/uncertainty/data/data_utils.py"

    DataLayer -->|"datasets"| GenerationScript

    GenerationScript["generate_answers.py"]:::script
    click GenerationScript "https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/generate_answers.py"

    subgraph "Model Abstraction Layer"
        BaseModel["BaseModel"]:::module
        HugModel["HuggingFaceModels"]:::module
        OpenAIAdapter["OpenAI Adapter"]:::module
        click BaseModel "https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/uncertainty/models/base_model.py"
        click HugModel "https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/uncertainty/models/huggingface_models.py"
        click OpenAIAdapter "https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/uncertainty/utils/openai.py"
    end

    GenerationScript --> BaseModel
    GenerationScript --> HugModel
    GenerationScript --> OpenAIAdapter
    HuggingFace --> HugModel
    OpenAI --> OpenAIAdapter
    GenerationScript -->|"JSON responses & hidden states"| RawOutputs

    RawOutputs["Raw Outputs\n(responses + hidden states)"]:::data
    RawOutputs -->|"raw outputs"| ComputeScript

    ComputeScript["compute_uncertainty_measures.py"]:::script
    click ComputeScript "https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/compute_uncertainty_measures.py"

    subgraph "Uncertainty Measures"
        p_ik["p_ik"]:::module
        p_true["p_true"]:::module
        semEnt["semantic_entropy"]:::module
        click p_ik "https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/uncertainty/uncertainty_measures/p_ik.py"
        click p_true "https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/uncertainty/uncertainty_measures/p_true.py"
        click semEnt "https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/uncertainty/uncertainty_measures/semantic_entropy.py"
    end

    ComputeScript --> p_ik
    ComputeScript --> p_true
    ComputeScript --> semEnt
    ComputeScript -->|"uncertainty scores"| Metrics

    Metrics["Uncertainty Metrics"]:::data
    Metrics -->|"metrics"| AnalysisScript

    AnalysisScript["analyze_results.py"]:::script
    click AnalysisScript "https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/analyze_results.py"

    AnalysisScript --> EvalUtils
    click EvalUtils "https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/uncertainty/utils/eval_utils.py"
    AnalysisScript -->|"aggregate metrics"| Plots
    AnalysisScript --> WandB

    Plots["Plots & Aggregates"]:::data
    Plots --> Notebook

    Notebook["example_evaluation.ipynb"]:::script
    click Notebook "https://github.com/jlko/semantic_uncertainty/blob/master/notebooks/example_evaluation.ipynb"
    WandB -->|"logs & metrics"| Notebook

    subgraph "Project Files"
        Env["environment.yaml\nenvironment_export.yaml"]:::file
        Docs["README.md\nLICENSE"]:::file
        click Env "https://github.com/jlko/semantic_uncertainty/blob/master/environment.yaml"
        click Env "https://github.com/jlko/semantic_uncertainty/blob/master/environment_export.yaml"
        click Docs "https://github.com/jlko/semantic_uncertainty/blob/master/README.md"
        click Docs "https://github.com/jlko/semantic_uncertainty/tree/master/LICENSE"
    end

    %% Styles
    classDef script fill:#e5ffe5,stroke:#2d572c,stroke-width:1px;
    classDef module fill:#ddeeff,stroke:#1f3a93,stroke-width:1px;
    classDef service fill:#ffe5cc,stroke:#b45f06,stroke-width:1px;
    classDef data fill:#e5e5ff,stroke:#2c3e50,stroke-width:1px;
    classDef file fill:#f0f0f0,stroke:#636e72,stroke-width:1px;
```
