# sajjadeslamkhah/llm-amr-benchmark architecture

> Architecture diagram of the GitHub repository sajjadeslamkhah/llm-amr-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sajjadeslamkhah/llm-amr-benchmark
- Repository: https://github.com/sajjadeslamkhah/llm-amr-benchmark

## Overview

This repository is a research benchmark comparing Claude, Gemini, and GPT agents, plus an AMRFinderPlus reference arm, on predicting E. coli antimicrobial resistance from genomes. Its workflow obtains phenotype records, selects isolates, fetches and anonymizes genomes, generates predictions, and compares them with laboratory phenotypes. Separate analyses assess hallucination, reproducibility, uncertainty, sensitivity, and run provenance. The README documents the main data-preparation workflow; sampled scripts show prediction and analysis details. Some secondary-analysis wiring is not shown, so those links are omitted.

Main entry points: [`scripts/05_run_agent.py`](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/05_run_agent.py) (Agent runner), [`scripts/01_fetch_amr.py`](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/01_fetch_amr.py) (Phenotype fetch), [`scripts/metrics.py`](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/metrics.py) (Prediction metrics), [`scripts/02_select_isolates.py`](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/02_select_isolates.py) (Isolate selection)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Study data preparation"]
  node_phenotypes["Phenotype fetch<br/>[01_fetch_amr.py]"]
  node_selection["Isolate selection"]
  node_genomes["Genome retrieval"]
end

subgraph group_arms["Prediction arms"]
  node_reference["AMRFinderPlus arm"]
  node_prompt["Agent task prompt<br/>[task_prompt.md]"]
  node_runner["Agent runner<br/>[05_run_agent.py]"]
end

subgraph group_review["Audit and analysis"]
  node_event_normalize["Event normalization<br/>[events_to_audit.py]"]
  node_hallucination["Hallucination audit"]
  node_metrics["Prediction metrics<br/>[metrics.py]"]
  node_repro["Reproducibility analysis<br/>[reproducibility.py]"]
  node_sensitivity["Sensitivity analysis<br/>[sensitivity.py]"]
  node_uncertainty["Uncertainty analysis<br/>[uncertainty.py]"]
  node_provenance["Run provenance<br/>[run_provenance.py]"]
  node_composition["Composition analysis"]
end

subgraph group_report["Study outputs"]
  node_manuscript["Manuscript document builder"]
  node_pdf["Markdown PDF renderer<br/>[md_to_pdf.py]"]
end

node_researcher(("Researcher"))
node_bvbrc["BV-BRC"]
node_amrfinder["AMRFinderPlus"]
node_agent["LLM agents"]

node_researcher -->|"starts workflow"| node_phenotypes
node_bvbrc -->|"supplies records"| node_phenotypes
node_phenotypes -->|"feeds selection"| node_selection
node_selection -->|"selects isolates"| node_genomes
node_genomes -->|"provides assemblies"| node_reference
node_amrfinder -->|"provides calls"| node_reference
node_prompt -->|"guides runs"| node_runner
node_agent -->|"returns predictions"| node_runner
node_runner -->|"writes predictions"| node_metrics
node_event_normalize -->|"normalizes events"| node_hallucination
node_runner -.->|"records events"| node_event_normalize
node_runner -.->|"supplies run logs"| node_hallucination
node_researcher -->|"scores benchmark"| node_metrics
node_metrics -->|"shares evaluation logic"| node_sensitivity
node_researcher -->|"checks repeatability"| node_repro
node_researcher -.->|"examines confidence"| node_uncertainty
node_researcher -.->|"records run context"| node_provenance
node_researcher -.->|"tests run composition"| node_composition

click node_phenotypes "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/01_fetch_amr.py"
click node_selection "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/02_select_isolates.py"
click node_genomes "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/03_fetch_genomes.py"
click node_reference "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/04_reference_arm.py"
click node_prompt "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/prompts/task_prompt.md"
click node_runner "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/05_run_agent.py"
click node_event_normalize "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/events_to_audit.py"
click node_hallucination "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/hallucination_audit.py"
click node_metrics "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/metrics.py"
click node_repro "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/reproducibility.py"
click node_sensitivity "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/sensitivity.py"
click node_uncertainty "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/uncertainty.py"
click node_provenance "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/run_provenance.py"
click node_composition "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/kompozisyon_analiz.py"
click node_manuscript "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/build_intro_methods_docx.js"
click node_pdf "https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/md_to_pdf.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_phenotypes,node_selection,node_genomes toneBlue
class node_reference,node_prompt,node_runner toneAmber
class node_event_normalize,node_hallucination,node_metrics,node_repro,node_sensitivity,node_uncertainty,node_provenance,node_composition toneMint
class node_manuscript,node_pdf toneRose
class node_researcher,node_bvbrc,node_amrfinder,node_agent toneIndigo
```

## Components

### Study data preparation

- [Phenotype fetch](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/01_fetch_amr.py): `scripts/01_fetch_amr.py`
- [Isolate selection](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/02_select_isolates.py): `scripts/02_select_isolates.py`
- [Genome retrieval](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/03_fetch_genomes.py): `scripts/03_fetch_genomes.py`

### Prediction arms

- [AMRFinderPlus arm](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/04_reference_arm.py): `scripts/04_reference_arm.py`
- [Agent task prompt](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/prompts/task_prompt.md): `prompts/task_prompt.md`
- [Agent runner](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/05_run_agent.py): `scripts/05_run_agent.py`

### Audit and analysis

- [Event normalization](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/events_to_audit.py): `scripts/events_to_audit.py`
- [Hallucination audit](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/hallucination_audit.py): `scripts/hallucination_audit.py`
- [Prediction metrics](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/metrics.py): `scripts/metrics.py`
- [Reproducibility analysis](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/reproducibility.py): `scripts/reproducibility.py`
- [Sensitivity analysis](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/sensitivity.py): `scripts/sensitivity.py`
- [Uncertainty analysis](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/uncertainty.py): `scripts/uncertainty.py`
- [Run provenance](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/run_provenance.py): `scripts/run_provenance.py`
- [Composition analysis](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/kompozisyon_analiz.py): `scripts/kompozisyon_analiz.py`

### Study outputs

- [Manuscript document builder](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/build_intro_methods_docx.js): `scripts/build_intro_methods_docx.js`
- [Markdown PDF renderer](https://github.com/sajjadeslamkhah/llm-amr-benchmark/blob/main/scripts/md_to_pdf.py): `scripts/md_to_pdf.py`

### Other components

- **Researcher**
- **BV-BRC**
- **AMRFinderPlus**
- **LLM agents**

## Connections

- Researcher → Phenotype fetch: starts workflow (evidence: `README.md`)
- BV-BRC → Phenotype fetch: supplies records (evidence: `README.md`)
- Phenotype fetch → Isolate selection: feeds selection (evidence: `README.md`)
- Isolate selection → Genome retrieval: selects isolates (evidence: `README.md`)
- Genome retrieval → AMRFinderPlus arm: provides assemblies (evidence: `README.md`)
- AMRFinderPlus → AMRFinderPlus arm: provides calls (evidence: `README.md`)
- Agent task prompt → Agent runner: guides runs (evidence: `README.md`)
- LLM agents → Agent runner: returns predictions (evidence: `scripts/05_run_agent.py`)
- Agent runner → Prediction metrics: writes predictions (evidence: `scripts/05_run_agent.py`)
- Event normalization → Hallucination audit: normalizes events (evidence: `scripts/events_to_audit.py`)
- Agent runner → Event normalization: records events
- Agent runner → Hallucination audit: supplies run logs
- Researcher → Prediction metrics: scores benchmark (evidence: `README.md`)
- Prediction metrics → Sensitivity analysis: shares evaluation logic (evidence: `scripts/sensitivity.py`)
- Researcher → Reproducibility analysis: checks repeatability (evidence: `README.md`)
- Researcher → Uncertainty analysis: examines confidence
- Researcher → Run provenance: records run context
- Researcher → Composition analysis: tests run composition
