How ethz-spylab/agentdojo works
AgentDojo is a Python benchmark for evaluating prompt-injection attacks and defenses in LLM agents. A researcher selects a benchmark suite, tasks, model, attack, and defense; the agent pipeline processes the task and invokes simulated tools against a suite environment. Task definitions provide ground-truth calls and evaluate task utility or injection security from outputs and environment state. The README documents these benchmark capabilities; sampled source confirms pipeline execution and task evaluation contracts, but not the full benchmark runner wiring.
Main entry points: agent_pipeline.py Agent pipeline · base_tasks.py Utility and security checks · default_suites Suite definitions · functions_runtime.py Functions runtime
Components 18
Agent orchestration
- Agent pipeline
src/agentdojo/agent_pipeline/agent_pipeline.py - Pipeline elements
src/agentdojo/agent_pipeline/basic_elements.py - LLM adapters
src/agentdojo/agent_pipeline/llms - Tool execution
src/agentdojo/agent_pipeline/tool_execution.py
Benchmark tasks
- Suite loader
src/agentdojo/task_suite/load_suites.py - Suite registry
src/agentdojo/task_suite/task_suite.py - User tasks
src/agentdojo/base_tasks.py - Injection tasks
src/agentdojo/base_tasks.py - Suite definitions
src/agentdojo/default_suites
Simulated environment
- Functions runtime
src/agentdojo/functions_runtime.py - Simulated tools
src/agentdojo/default_suites/v1/tools - Suite environments
src/agentdojo/default_suites/v1
Attack and defense
- Attack strategies
src/agentdojo/attacks - Injection detector
src/agentdojo/agent_pipeline/pi_detector.py
Evaluation
- Ground-truth pipeline
src/agentdojo/agent_pipeline/ground_truth_pipeline.py - Utility and security checks
src/agentdojo/base_tasks.py
Other components
- Researcher
- Model provider
Connections 22
- Researcher → Suite loader: selects suite
- Suite loader → Suite registry: loads suites
- Suite registry → Suite definitions: registers tasks
- Suite definitions → User tasks: defines tasks
- Suite definitions → Injection tasks: defines attacks
- Researcher → Agent pipeline: configures run
- Agent pipeline → Pipeline elements: runs elements
- Agent pipeline → LLM adapters: selects model
- LLM adapters → Model provider: requests completion
- Pipeline elements → Tool execution: dispatches calls
- Tool execution → Functions runtime: runs function
- Functions runtime → Simulated tools: dispatches tool
- Simulated tools → Suite environments: reads and updates
- Researcher → Attack strategies: selects attack
- Attack strategies → Agent pipeline: supplies attack input
- Agent pipeline → Injection detector: applies defense
- Ground-truth pipeline → User tasks: gets expected calls
- Ground-truth pipeline → Functions runtime: executes expected calls
- Agent pipeline → Utility and security checks: provides output
- User tasks → Utility and security checks: checks utility
- Injection tasks → Utility and security checks: checks security
- Utility and security checks → Suite environments: compares state