# xiheng724/llm-dependency-benchmark architecture

> Architecture diagram of the GitHub repository xiheng724/llm-dependency-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xiheng724/llm-dependency-benchmark
- Repository: https://github.com/xiheng724/llm-dependency-benchmark

## Overview

LLM Dependency Benchmark converts agent traces into cases, obtains human dependency annotations, queries remote models to attribute outputs to input spans, and scores predictions against gold. Its experiment suite also covers baselines, on-policy behavior, judge robustness, and injection detection. The README documents those product stages; sampled source confirms parts of AgentDojo conversion, rendering, evaluation, annotation export, and analysis, so unsampled subsystem links are limited to documented behavior.

Main entry points: [`eval/evaluate.py`](https://github.com/xiheng724/llm-dependency-benchmark/blob/main/eval/evaluate.py) (Evaluation entry point), [`tools/human_gold/human_gold_export.py`](https://github.com/xiheng724/llm-dependency-benchmark/blob/main/tools/human_gold/human_gold_export.py) (Annotation export), [`eval/providers.py`](https://github.com/xiheng724/llm-dependency-benchmark/blob/main/eval/providers.py) (Provider clients), [`eval/baselines.py`](https://github.com/xiheng724/llm-dependency-benchmark/blob/main/eval/baselines.py) (Offline baselines)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Cases and conversion"]
  node_framework_adapters["Framework adapters"]
  node_agentdojo_converter["AgentDojo conversion<br/>[convert.py]"]
  node_case_core["Case structures"]
  node_dataset_tools["Dataset construction"]
  node_case_inputs["Dependency cases"]
end

subgraph group_gold["Human gold"]
  node_gold_export["Annotation export"]
  node_gold_staging["Gold staging"]
  node_gold_workflow["Adjudication and overlays"]
  node_gold_data["Human gold overlays"]
end

subgraph group_eval["Evaluation"]
  node_evaluation["Evaluation entry point<br/>[evaluate.py]"]
  node_payload_render["Tagged transcript rendering<br/>[payload_render.py]"]
  node_provider_clients["Provider clients<br/>[providers.py]"]
  node_scoring["Scoring and taxonomy<br/>[scoring.py]"]
  node_baseline["Offline baselines<br/>[baselines.py]"]
  node_results["Predictions and scores"]
end

subgraph group_experiments["Experiments"]
  node_rollouts["On-policy rollouts<br/>[rollout_ab.py]"]
  node_experiment_drivers["Experiment drivers"]
  node_analysis["Experiment analysis"]
end

node_researcher(("Researcher"))
node_annotators(("Annotators"))
node_provider{{"Model provider"}}

node_researcher -->|"selects traces"| node_framework_adapters
node_framework_adapters -->|"converts traces"| node_case_inputs
node_agentdojo_converter -->|"replays and records"| node_case_inputs
node_case_core -->|"supports conversion"| node_agentdojo_converter
node_researcher -->|"constructs variants"| node_dataset_tools
node_dataset_tools -->|"derives cases"| node_case_inputs
node_researcher -->|"selects cases"| node_gold_export
node_gold_export -->|"reads cases"| node_case_inputs
node_gold_export -->|"exports items"| node_annotators
node_annotators -->|"supply judgments"| node_gold_workflow
node_gold_workflow -->|"exports overlays"| node_gold_data
node_gold_staging -->|"stages annotations"| node_gold_data
node_researcher -->|"runs evaluation"| node_evaluation
node_evaluation -->|"reads cases"| node_case_inputs
node_evaluation -->|"loads gold"| node_gold_data
node_evaluation -->|"renders inputs"| node_payload_render
node_evaluation -->|"requests predictions"| node_provider_clients
node_provider_clients -.->|"calls API"| node_provider
node_evaluation -->|"scores predictions"| node_scoring
node_evaluation -->|"writes results"| node_results
node_baseline -->|"compares candidates"| node_case_inputs
node_baseline -->|"writes scores"| node_results
node_experiment_drivers -->|"launches runs"| node_evaluation
node_rollouts -->|"requests actions"| node_provider_clients
node_rollouts -->|"records rollouts"| node_results
node_analysis -->|"analyzes outputs"| node_results
node_analysis -->|"compares gold"| node_gold_data

click node_framework_adapters "https://github.com/xiheng724/llm-dependency-benchmark/tree/main/adapters"
click node_agentdojo_converter "https://github.com/xiheng724/llm-dependency-benchmark/blob/main/adapters/agentdojo/convert.py"
click node_case_core "https://github.com/xiheng724/llm-dependency-benchmark/tree/main/core"
click node_dataset_tools "https://github.com/xiheng724/llm-dependency-benchmark/tree/main/tools/datasets"
click node_gold_export "https://github.com/xiheng724/llm-dependency-benchmark/blob/main/tools/human_gold/human_gold_export.py"
click node_gold_staging "https://github.com/xiheng724/llm-dependency-benchmark/blob/main/adapters/human_gold_staging.py"
click node_gold_workflow "https://github.com/xiheng724/llm-dependency-benchmark/tree/main/tools/human_gold"
click node_evaluation "https://github.com/xiheng724/llm-dependency-benchmark/blob/main/eval/evaluate.py"
click node_payload_render "https://github.com/xiheng724/llm-dependency-benchmark/blob/main/core/payload_render.py"
click node_provider_clients "https://github.com/xiheng724/llm-dependency-benchmark/blob/main/eval/providers.py"
click node_scoring "https://github.com/xiheng724/llm-dependency-benchmark/blob/main/eval/scoring.py"
click node_baseline "https://github.com/xiheng724/llm-dependency-benchmark/blob/main/eval/baselines.py"
click node_rollouts "https://github.com/xiheng724/llm-dependency-benchmark/blob/main/eval/rollout_ab.py"
click node_experiment_drivers "https://github.com/xiheng724/llm-dependency-benchmark/tree/main/tools/runtime"
click node_analysis "https://github.com/xiheng724/llm-dependency-benchmark/tree/main/tools/analysis"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_framework_adapters,node_agentdojo_converter,node_case_core,node_dataset_tools,node_case_inputs toneBlue
class node_gold_export,node_gold_staging,node_gold_workflow,node_gold_data toneAmber
class node_evaluation,node_payload_render,node_provider_clients,node_scoring,node_baseline,node_results toneMint
class node_rollouts,node_experiment_drivers,node_analysis toneRose
class node_researcher,node_annotators,node_provider toneIndigo
```

## Components

### Cases and conversion

- [Framework adapters](https://github.com/xiheng724/llm-dependency-benchmark/tree/main/adapters): `adapters`
- [AgentDojo conversion](https://github.com/xiheng724/llm-dependency-benchmark/blob/main/adapters/agentdojo/convert.py): `adapters/agentdojo/convert.py`
- [Case structures](https://github.com/xiheng724/llm-dependency-benchmark/tree/main/core): `core`
- [Dataset construction](https://github.com/xiheng724/llm-dependency-benchmark/tree/main/tools/datasets): `tools/datasets`
- **Dependency cases**

### Human gold

- [Annotation export](https://github.com/xiheng724/llm-dependency-benchmark/blob/main/tools/human_gold/human_gold_export.py): `tools/human_gold/human_gold_export.py`
- [Gold staging](https://github.com/xiheng724/llm-dependency-benchmark/blob/main/adapters/human_gold_staging.py): `adapters/human_gold_staging.py`
- [Adjudication and overlays](https://github.com/xiheng724/llm-dependency-benchmark/tree/main/tools/human_gold): `tools/human_gold`
- **Human gold overlays**

### Evaluation

- [Evaluation entry point](https://github.com/xiheng724/llm-dependency-benchmark/blob/main/eval/evaluate.py): `eval/evaluate.py`
- [Tagged transcript rendering](https://github.com/xiheng724/llm-dependency-benchmark/blob/main/core/payload_render.py): `core/payload_render.py`
- [Provider clients](https://github.com/xiheng724/llm-dependency-benchmark/blob/main/eval/providers.py): `eval/providers.py`
- [Scoring and taxonomy](https://github.com/xiheng724/llm-dependency-benchmark/blob/main/eval/scoring.py): `eval/scoring.py`
- [Offline baselines](https://github.com/xiheng724/llm-dependency-benchmark/blob/main/eval/baselines.py): `eval/baselines.py`
- **Predictions and scores**

### Experiments

- [On-policy rollouts](https://github.com/xiheng724/llm-dependency-benchmark/blob/main/eval/rollout_ab.py): `eval/rollout_ab.py`
- [Experiment drivers](https://github.com/xiheng724/llm-dependency-benchmark/tree/main/tools/runtime): `tools/runtime`
- [Experiment analysis](https://github.com/xiheng724/llm-dependency-benchmark/tree/main/tools/analysis): `tools/analysis`

### Other components

- **Researcher**
- **Annotators**
- **Model provider**

## Connections

- Researcher → Framework adapters: selects traces
- Framework adapters → Dependency cases: converts traces
- AgentDojo conversion → Dependency cases: replays and records
- Case structures → AgentDojo conversion: supports conversion
- Researcher → Dataset construction: constructs variants
- Dataset construction → Dependency cases: derives cases
- Researcher → Annotation export: selects cases
- Annotation export → Dependency cases: reads cases
- Annotation export → Annotators: exports items
- Annotators → Adjudication and overlays: supply judgments
- Adjudication and overlays → Human gold overlays: exports overlays
- Gold staging → Human gold overlays: stages annotations
- Researcher → Evaluation entry point: runs evaluation
- Evaluation entry point → Dependency cases: reads cases
- Evaluation entry point → Human gold overlays: loads gold
- Evaluation entry point → Tagged transcript rendering: renders inputs
- Evaluation entry point → Provider clients: requests predictions
- Provider clients → Model provider: calls API
- Evaluation entry point → Scoring and taxonomy: scores predictions
- Evaluation entry point → Predictions and scores: writes results
- Offline baselines → Dependency cases: compares candidates
- Offline baselines → Predictions and scores: writes scores
- Experiment drivers → Evaluation entry point: launches runs
- On-policy rollouts → Provider clients: requests actions
- On-policy rollouts → Predictions and scores: records rollouts
- Experiment analysis → Predictions and scores: analyzes outputs
- Experiment analysis → Human gold overlays: compares gold