How mli0118/ontology-llm works
This repository implements a benchmark for detecting taxonomic hallucinations in a language model, comparing an Algebraic Ontology Projection (AOP) detector with confidence and consistency baselines. The intended workflow is to derive labeled probes from an ontology, obtain model answers and hidden-state representations, evaluate detectors, and report results. The sampled modules establish ontology-to-probe generation, model I/O, AOP training/evaluation, and baseline calls. benchmark.py and the result artifact are present in the tree but were not sampled, so their orchestration and data links are not asserted.
Main entry points: model_io.py Model I/O · probes.py Probe generation · aop.py AOP projection · baselines.py Baseline detectors
Components 16
Benchmark data
- Taxonomic ontology
files/ontology.py - Probe generation
files/probes.py - Labeled probes
Model access
- Model I/O
files/model_io.py - Causal language model
- Concept representations
- Yes/no judgments
Detection methods
- AOP projection
files/aop.py - Projection training
files/aop.py - Code evaluation
files/aop.py - Baseline detectors
files/baselines.py - Confidence baseline
files/baselines.py - Consistency baseline
files/baselines.py
Benchmark evaluation
- Benchmark runner
files/benchmark.py - Hallucination report
files/results/hallucination_report.json
Other components
- Researcher
Connections 13
- Researcher → Benchmark runner: initiates
- Probe generation → Taxonomic ontology: queries relations
- Probe generation → Labeled probes: generates
- Model I/O → Causal language model: loads and queries
- Model I/O → Concept representations: extracts
- Model I/O → Yes/no judgments: computes
- AOP projection → Projection training: trains
- Projection training → Concept representations: reads
- AOP projection → Code evaluation: evaluates codes
- Baseline detectors → Confidence baseline: defines
- Baseline detectors → Consistency baseline: defines
- Confidence baseline → Model I/O: requests confidence
- Consistency baseline → Model I/O: requests judgments