How skygod07/ai-agent-security---multi-step-tool-attacks works
This repository combines a Kaggle AI-agent security submission pipeline with a reusable SDK for attack algorithms, sandboxed tool interactions, guardrails, and evaluation. The main workflow is to implement or discover multi-step attack candidates, run them through an agent and tool environment, then score and inspect results through evaluation commands or the Kaggle gateway. It also contains a separate ROS visual-editor application, omitted here because the README identifies the competition pipeline as the product. Some subsystem links reflect documented capability or source-index references; unsampled implementation details are not inferred.
Main entry points: ops.py Evaluation Operations · jed_attack_gateway.py Competition Gateway · search.py Search Strategies · candidate.py Candidate Model
Components 24
Attack Discovery
- Search Strategies
our_work/src/attack_discovery/search.py - Candidate Model
our_work/src/attack_discovery/candidate.py - Candidate Mutation
our_work/src/attack_discovery/mutation.py - Trace Analysis
our_work/src/attack_discovery/trace.py - Attack Search Engine
our_work/src/attack_discovery/stockfish_engine.py
Agent Runtime
- Attack Contracts
our_work/aicomp_sdk/attacks/contracts.py - Agent Backends
our_work/aicomp_sdk/agents - Tool Suite
our_work/aicomp_sdk/core/tools - Trace Cells
our_work/aicomp_sdk/core/cells.py - Security Predicates
our_work/aicomp_sdk/core/predicates.py - Guardrails
our_work/aicomp_sdk/guardrails/optimal.py - Runtime Hooks
our_work/aicomp_sdk/hooks
Evaluation
- Evaluation Operations
our_work/aicomp_sdk/evaluation/ops.py - Run Diagnostics
our_work/aicomp_sdk/evaluation/diagnostics - Scoring
our_work/aicomp_sdk/scoring.py
Submission Interface
- SDK CLI
our_work/aicomp_sdk/cli/main.py - Evaluate Command
our_work/aicomp_sdk/cli/commands/evaluate.py - Competition Gateway
our_work/kaggle_evaluation/jed_attack_134815/jed_attack_gateway.py - Inference Server
our_work/kaggle_evaluation/jed_attack_134815/jed_attack_inference_server.py - Remote Agent
our_work/kaggle_evaluation/jed_attack_134815/remote_agent.py - Gateway Base
our_work/kaggle_evaluation/core/base_gateway.py - Evaluation Relay
our_work/kaggle_evaluation/core/relay.py - Gateway Templates
our_work/kaggle_evaluation/core/templates.py
Other components
- Researcher
Connections 25
- Researcher → SDK CLI: runs commands
our_work/aicomp_sdk/cli/main.py - SDK CLI → Evaluate Command: dispatches evaluation
our_work/aicomp_sdk/cli/main.py - Evaluate Command → Evaluation Operations: runs evaluation
our_work/aicomp_sdk/cli/commands/evaluate.py - Evaluation Operations → Attack Contracts: loads attack algorithms
our_work/aicomp_sdk/evaluation/ops.py - Evaluation Operations → Security Predicates: checks traces
our_work/aicomp_sdk/evaluation/ops.py - Evaluation Operations → Trace Cells: signs tool events
our_work/aicomp_sdk/evaluation/ops.py - Evaluation Operations → Guardrails: uses guardrails
our_work/aicomp_sdk/evaluation/ops.py - Evaluation Operations → Runtime Hooks: loads hooks
our_work/aicomp_sdk/evaluation/ops.py - Evaluation Operations → Scoring: scores findings
our_work/aicomp_sdk/evaluation/ops.py - Evaluation Operations → Run Diagnostics: emits diagnostics
our_work/aicomp_sdk/evaluation/ops.py - Search Strategies → Candidate Model: evaluates candidates
our_work/src/attack_discovery/search.py - Search Strategies → Candidate Mutation: explores mutations
our_work/src/attack_discovery/search.py - Search Strategies → Trace Analysis: analyzes traces
our_work/src/attack_discovery/search.py - Candidate Model → Attack Contracts: uses attack contracts
our_work/src/attack_discovery/candidate.py - Candidate Model → Security Predicates: checks predicates
our_work/src/attack_discovery/candidate.py - Attack Search Engine → Trace Analysis: evaluates traces
our_work/src/attack_discovery/stockfish_engine.py - Competition Gateway → Gateway Base: extends gateway
our_work/kaggle_evaluation/jed_attack_134815/jed_attack_gateway.py - Competition Gateway → Evaluation Operations: runs evaluation
our_work/kaggle_evaluation/jed_attack_134815/jed_attack_gateway.py - Competition Gateway → Agent Backends: uses agent backends
our_work/kaggle_evaluation/jed_attack_134815/jed_attack_gateway.py - Competition Gateway → Guardrails: uses guardrails
our_work/kaggle_evaluation/jed_attack_134815/jed_attack_gateway.py - Competition Gateway → Remote Agent: dispatches agent
our_work/kaggle_evaluation/jed_attack_134815/jed_attack_gateway.py - Competition Gateway → Gateway Templates: uses templates
our_work/kaggle_evaluation/jed_attack_134815/jed_attack_gateway.py - Inference Server → Competition Gateway: uses gateway
our_work/kaggle_evaluation/jed_attack_134815/jed_attack_inference_server.py - Inference Server → Evaluation Operations: runs evaluation
our_work/kaggle_evaluation/jed_attack_134815/jed_attack_inference_server.py - Gateway Base → Evaluation Relay: uses relay
our_work/kaggle_evaluation/core/base_gateway.py