How sajjadeslamkhah/llm-amr-benchmark works
This repository is a research benchmark comparing Claude, Gemini, and GPT agents, plus an AMRFinderPlus reference arm, on predicting E. coli antimicrobial resistance from genomes. Its workflow obtains phenotype records, selects isolates, fetches and anonymizes genomes, generates predictions, and compares them with laboratory phenotypes. Separate analyses assess hallucination, reproducibility, uncertainty, sensitivity, and run provenance. The README documents the main data-preparation workflow; sampled scripts show prediction and analysis details. Some secondary-analysis wiring is not shown, so those links are omitted.
Main entry points: 05_run_agent.py Agent runner · 01_fetch_amr.py Phenotype fetch · metrics.py Prediction metrics · 02_select_isolates.py Isolate selection
Components 20
Study data preparation
- Phenotype fetch
scripts/01_fetch_amr.py - Isolate selection
scripts/02_select_isolates.py - Genome retrieval
scripts/03_fetch_genomes.py
Prediction arms
- AMRFinderPlus arm
scripts/04_reference_arm.py - Agent task prompt
prompts/task_prompt.md - Agent runner
scripts/05_run_agent.py
Audit and analysis
- Event normalization
scripts/events_to_audit.py - Hallucination audit
scripts/hallucination_audit.py - Prediction metrics
scripts/metrics.py - Reproducibility analysis
scripts/reproducibility.py - Sensitivity analysis
scripts/sensitivity.py - Uncertainty analysis
scripts/uncertainty.py - Run provenance
scripts/run_provenance.py - Composition analysis
scripts/kompozisyon_analiz.py
Study outputs
- Manuscript document builder
scripts/build_intro_methods_docx.js - Markdown PDF renderer
scripts/md_to_pdf.py
Other components
- Researcher
- BV-BRC
- AMRFinderPlus
- LLM agents
Connections 18
- Researcher → Phenotype fetch: starts workflow
README.md - BV-BRC → Phenotype fetch: supplies records
README.md - Phenotype fetch → Isolate selection: feeds selection
README.md - Isolate selection → Genome retrieval: selects isolates
README.md - Genome retrieval → AMRFinderPlus arm: provides assemblies
README.md - AMRFinderPlus → AMRFinderPlus arm: provides calls
README.md - Agent task prompt → Agent runner: guides runs
README.md - LLM agents → Agent runner: returns predictions
scripts/05_run_agent.py - Agent runner → Prediction metrics: writes predictions
scripts/05_run_agent.py - Event normalization → Hallucination audit: normalizes events
scripts/events_to_audit.py - Agent runner → Event normalization: records events
- Agent runner → Hallucination audit: supplies run logs
- Researcher → Prediction metrics: scores benchmark
README.md - Prediction metrics → Sensitivity analysis: shares evaluation logic
scripts/sensitivity.py - Researcher → Reproducibility analysis: checks repeatability
README.md - Researcher → Uncertainty analysis: examines confidence
- Researcher → Run provenance: records run context
- Researcher → Composition analysis: tests run composition