How zixin22/fragfuse works
FragFuse is a research suite for evaluating memory-based access-control risks in agents, organized into AgentHarm, RAP/WebShop, and OSAgent/SeeAct experiment tracks. The main workflow constructs or injects fragmented intent, runs an agent in a benchmark environment, and applies rule or guard-agent checks to assess unsafe behavior. The graph preserves the tracks’ distinct implementations rather than implying they share a runtime. Relationships are limited to README statements and source-index references or excerpts; several pipeline stages are documented but their internal calls were not sampled.
Main entry points: main.py Attack runner · webshop_guard_agent.py WebShop guard adapter · agentharm.py Benchmark task · guardrail.py Action guardrail
Components 34
AgentHarm evaluation
- Benchmark task
Agentharm/src/inspect_evals/agentharm/agentharm.py - Agent variants
Agentharm/src/inspect_evals/agentharm/agents - Memory agent
Agentharm/src/inspect_evals/agentharm/agents/memory_agent.py - Fragment extraction
Agentharm/src/inspect_evals/agentharm/fragfuse_pipeline/1extract_fragments.py - Iterative masking
Agentharm/src/inspect_evals/agentharm/fragfuse_pipeline/2_1iterative_mask.py - Fragment summaries
Agentharm/src/inspect_evals/agentharm/fragfuse_pipeline - Host checks
Agentharm/src/inspect_evals/agentharm/fragfuse_pipeline/4host_check.py - Memory injection
Agentharm/src/inspect_evals/agentharm/fragfuse_pipeline/5injection_store.py - Retrieval tests
Agentharm/src/inspect_evals/agentharm/fragfuse_pipeline/8retrieve_rate.py - Guardrail prompts
Agentharm/src/inspect_evals/agentharm/guardrail_prompt/guardrail_prompt.py - Evaluation scoring
Agentharm/src/inspect_evals/agentharm/scorer.py
RAP WebShop
- Attack runner
RAP/webshop/main.py - Fragment construction
RAP/webshop/fragExtracor_pipeline/frag_mask_pipeline.py - WebShop prompts
RAP/webshop/prompts/webshop_prompt.py - Rule checking
RAP/webshop/rule_and_profile/rule_checker.py - User profiles
RAP/webshop/rule_and_profile/user_profile.py - Rule examples
RAP/webshop/rule_and_profile/request_webshop.py - Rule metrics
RAP/webshop/rule_and_profile/metrics.py - WebShop guard adapter
RAP/guard agent/webshop_guard_agent.py - Guard agent
RAP/guard agent/guardagent.py - Guard prompts
RAP/guard agent/prompts_guard.py - WebShop guard tools
RAP/guard agent/toolset_webshop.py - Query optimization
RAP/webshop/optimization/optimization_webshop - Carrier templates
RAP/webshop/optimization/optimization_webshop/carrier_query_template_utils.py
OSAgent and SeeAct
- OS experiment runner
OSAgent and SeeAct/AGrail4Agent/DAS/exp_OS.py - Action guardrail
OSAgent and SeeAct/AGrail4Agent/DAS/guardrail.py - Guard prompts
OSAgent and SeeAct/AGrail4Agent/DAS/prompts.py - Permission detection
OSAgent and SeeAct/AGrail4Agent/DAS/tools/permission_tool.py - Tool interface
OSAgent and SeeAct/AGrail4Agent/DAS/tools/tool.py - Web detection
OSAgent and SeeAct/AGrail4Agent/DAS/tools/web_tool.py - Code detection
OSAgent and SeeAct/AGrail4Agent/DAS/tools/code_tool.py
Other components
- Researcher
- Benchmark operator
- Experiment operator
Connections 40
- Researcher → Benchmark task: configures
Agentharm/src/inspect_evals/agentharm/agentharm.py - Benchmark task → Agent variants: selects agent
Agentharm/src/inspect_evals/agentharm/agentharm.py - Memory agent → Agent variants: implements variant
Agentharm/src/inspect_evals/agentharm/agents/agent.py - Memory agent → Benchmark task: runs in task
- Benchmark task → Evaluation scoring: scores results
Agentharm/src/inspect_evals/agentharm/agentharm.py - Evaluation scoring → Benchmark task: uses metrics
Agentharm/src/inspect_evals/agentharm/scorer.py - Memory agent → Guardrail prompts: uses prompts
Agentharm/src/inspect_evals/agentharm/agents/memory_agent.py - Researcher → Fragment extraction: starts pipeline
README.md - Fragment extraction → Iterative masking: precedes masking
- Iterative masking → Fragment summaries: prepares fragments
- Fragment summaries → Host checks: feeds host checks
- Host checks → Memory injection: precedes injection
- Memory injection → Retrieval tests: followed by retrieval
README.md - Retrieval tests → Guardrail prompts: evaluates mitigation
- Benchmark operator → Attack runner: runs experiment
README.md - Attack runner → Fragment construction: uses attack inputs
- Attack runner → WebShop prompts: uses prompts
RAP/webshop/main.py - Attack runner → Rule checking: runs rule checks
RAP/webshop/main.py - Attack runner → WebShop guard adapter: selects defense
RAP/webshop/main.py - Attack runner → Rule metrics: evaluates outcomes
- Rule checking → User profiles: checks profile
RAP/webshop/rule_and_profile/rule_checker.py - Rule checking → Rule examples: uses rule examples
- WebShop guard adapter → Guard agent: delegates guard work
RAP/guard agent/webshop_guard_agent.py - WebShop guard adapter → WebShop guard tools: uses toolset
RAP/guard agent/webshop_guard_agent.py - WebShop guard adapter → User profiles: uses profile data
RAP/guard agent/webshop_guard_agent.py - WebShop guard adapter → Rule examples: uses rule examples
RAP/guard agent/webshop_guard_agent.py - Guard agent → Guard prompts: loads prompts
RAP/guard agent/guardagent.py - WebShop guard tools → Guard prompts: uses prompt definitions
RAP/guard agent/toolset_webshop.py - Benchmark operator → Query optimization: optimizes queries
- Query optimization → Carrier templates: rewrites templates
RAP/webshop/optimization/optimization_webshop/evaluator.py - Fragment construction → Carrier templates: builds carrier queries
- Experiment operator → OS experiment runner: runs experiment
README.md - OS experiment runner → Guard prompts: uses prompts
OSAgent and SeeAct/AGrail4Agent/DAS/exp_OS.py - OS experiment runner → Action guardrail: invokes guardrail
OSAgent and SeeAct/AGrail4Agent/DAS/exp_OS.py - Action guardrail → Permission detection: dispatches checks
OSAgent and SeeAct/AGrail4Agent/DAS/guardrail.py - Action guardrail → Web detection: dispatches checks
OSAgent and SeeAct/AGrail4Agent/DAS/guardrail.py - Action guardrail → Code detection: dispatches checks
OSAgent and SeeAct/AGrail4Agent/DAS/guardrail.py - Permission detection → Tool interface: inherits interface
OSAgent and SeeAct/AGrail4Agent/DAS/tools/permission_tool.py - Web detection → Tool interface: inherits interface
OSAgent and SeeAct/AGrail4Agent/DAS/tools/web_tool.py - Code detection → Tool interface: inherits interface
OSAgent and SeeAct/AGrail4Agent/DAS/tools/code_tool.py