How xiheng724/llm-dependency-benchmark works
LLM Dependency Benchmark converts agent traces into cases, obtains human dependency annotations, queries remote models to attribute outputs to input spans, and scores predictions against gold. Its experiment suite also covers baselines, on-policy behavior, judge robustness, and injection detection. The README documents those product stages; sampled source confirms parts of AgentDojo conversion, rendering, evaluation, annotation export, and analysis, so unsampled subsystem links are limited to documented behavior.
Main entry points: evaluate.py Evaluation entry point · human_gold_export.py Annotation export · providers.py Provider clients · baselines.py Offline baselines
Components 21
Cases and conversion
- Framework adapters
adapters - AgentDojo conversion
adapters/agentdojo/convert.py - Case structures
core - Dataset construction
tools/datasets - Dependency cases
Human gold
- Annotation export
tools/human_gold/human_gold_export.py - Gold staging
adapters/human_gold_staging.py - Adjudication and overlays
tools/human_gold - Human gold overlays
Evaluation
- Evaluation entry point
eval/evaluate.py - Tagged transcript rendering
core/payload_render.py - Provider clients
eval/providers.py - Scoring and taxonomy
eval/scoring.py - Offline baselines
eval/baselines.py - Predictions and scores
Experiments
- On-policy rollouts
eval/rollout_ab.py - Experiment drivers
tools/runtime - Experiment analysis
tools/analysis
Other components
- Researcher
- Annotators
- Model provider
Connections 27
- Researcher → Framework adapters: selects traces
- Framework adapters → Dependency cases: converts traces
- AgentDojo conversion → Dependency cases: replays and records
- Case structures → AgentDojo conversion: supports conversion
- Researcher → Dataset construction: constructs variants
- Dataset construction → Dependency cases: derives cases
- Researcher → Annotation export: selects cases
- Annotation export → Dependency cases: reads cases
- Annotation export → Annotators: exports items
- Annotators → Adjudication and overlays: supply judgments
- Adjudication and overlays → Human gold overlays: exports overlays
- Gold staging → Human gold overlays: stages annotations
- Researcher → Evaluation entry point: runs evaluation
- Evaluation entry point → Dependency cases: reads cases
- Evaluation entry point → Human gold overlays: loads gold
- Evaluation entry point → Tagged transcript rendering: renders inputs
- Evaluation entry point → Provider clients: requests predictions
- Provider clients → Model provider: calls API
- Evaluation entry point → Scoring and taxonomy: scores predictions
- Evaluation entry point → Predictions and scores: writes results
- Offline baselines → Dependency cases: compares candidates
- Offline baselines → Predictions and scores: writes scores
- Experiment drivers → Evaluation entry point: launches runs
- On-policy rollouts → Provider clients: requests actions
- On-policy rollouts → Predictions and scores: records rollouts
- Experiment analysis → Predictions and scores: analyzes outputs
- Experiment analysis → Human gold overlays: compares gold