How ivicanikolicsg/model_provenance_testing works
This repository implements statistical model-provenance testing: it compares a candidate language model’s prompt responses with responses from possible parent models and reports a parent guess, optionally evaluating it against supplied ground truth. The main workflow is to load parent and candidate lists, reuse or generate parent prompt-output caches, query each candidate on those prompts, and statistically compare response-match rates. The graph includes the documented benchmark inputs and sampling options; relationships involving unsampled implementation details are omitted.
Main entry points: tester.py Test runner · cache_parent_prompts.py Parent cache builder · llm_functions.py LLM inference · sampler_classes.py Prompt samplers
Components 13
Test workflow
- Test runner
tester.py - Model lists
data
Prompt generation
- Prompt samplers
sampler_classes.py - Random sentences
data/random_sentences.txt - Parent cache builder
cache_parent_prompts.py - Prompt output cache
cached_prompts
Model inference
- LLM inference
llm_functions.py - Hugging Face Hub
Comparison and results
- Statistical tests
misc.py - Parent decision
tester.py - Ground-truth evaluation
tester.py - Test report
tester.py
Other components
- Researcher
Connections 15
- Researcher → Test runner: runs
README.md - Model lists → Test runner: provides models
README.md - Test runner → Prompt samplers: selects sampler
tester.py - Random sentences → Prompt samplers: provides prompts
README.md - Test runner → Parent cache builder: loads cache
tester.py - Parent cache builder → Prompt samplers: samples prompts
cache_parent_prompts.py - Parent cache builder → LLM inference: queries parents
cache_parent_prompts.py - Parent cache builder → Prompt output cache: writes outputs
cache_parent_prompts.py - Prompt output cache → Test runner: supplies prompts
- Test runner → LLM inference: queries candidates
tester.py - LLM inference → Hugging Face Hub: loads models
llm_functions.py - Test runner → Statistical tests: compares rates
tester.py - Test runner → Parent decision: selects parent
tester.py - Test runner → Ground-truth evaluation: scores guesses
tester.py - Test runner → Test report: prints results
tester.py