How antgroup/agent3sigma-canary works
Agent3σ-Canary evaluates AI agents by running realistic tool-using tasks in isolated environments and grading their execution trajectories. The main path is task selection and attack setup, sandboxed agent execution against simulated resources, then multidimensional scoring and result analysis. The README also documents multiple agent frameworks, safety tasks, attack strategies, and security-defense variants. Some orchestration and task-corpus source is not present in the supplied tree, so those links use supported directory paths; unsampled implementation details are omitted.
Main entry points: scripts Evaluation runner · lib_agent.py Agent execution · images Agent frameworks · lib_grading.py Trajectory grading
Components 22
Evaluation orchestration
- Evaluation runner
scripts - Task suites
scripts/lib_tasks.py - Attack methods
scripts/lib_attacks.py
Sandbox runtime
- Agent execution
scripts/lib_agent.py - Docker sandbox
scripts/lib_docker.py - Agent frameworks
workflow/images - Hermes shim
workflow/images/hermes/mock-api/app.py - NanoClaw shims
workflow/images/nanoclaw/shim/driver.mjs - OpenAI shim
workflow/images/nanoclaw_openai/shim/driver_openai.mjs
Defense variants
- Security defenses
workflow/images - OpenClaw Shield
workflow/images/official_shield/openclaw-shield/src/index.ts - SecureClaw
workflow/images/official_secureclaw/secureclaw/secureclaw/src/index.ts - ClawKeeper
workflow/images/official_clawkeeper/ClawKeeper/clawkeeper-plugin/src/index.js
Scoring and analysis
- Execution trajectory
scripts/lib_tracee.py - Trajectory grading
scripts/lib_grading.py - Trace correlation
scripts/lib_tracee_grading.py - Evaluation results
scripts - Results analyzer
workflow_analysis/app.py
Simulated resources
- Mock API
workflow/images/official/mock-api/app.py - Tool scenarios
_skills_repository
Other components
- Evaluator
- Auxiliary models
Connections 22
- Evaluator → Evaluation runner: starts evaluation
- Evaluation runner → Task suites: selects tasks
- Evaluation runner → Attack methods: configures attacks
- Evaluation runner → Docker sandbox: launches sandbox
- Evaluation runner → Agent execution: dispatches run
- Agent execution → Agent frameworks: runs agent
- Agent frameworks → Hermes shim: uses adapter
- Agent frameworks → NanoClaw shims: uses adapter
- Agent frameworks → OpenAI shim: uses adapter
- Agent execution → Mock API: calls simulated APIs
- Mock API → Tool scenarios: serves scenarios
- Agent execution → Execution trajectory: records execution
- Execution trajectory → Trace correlation: provides trace
- Evaluation runner → Trajectory grading: requests scoring
- Trajectory grading → Auxiliary models: requests judgments
- Trajectory grading → Evaluation results: writes scores
- Evaluation runner → Evaluation results: stores run data
- Results analyzer → Evaluation results: reads results
- Evaluation runner → Security defenses: selects variant
- Security defenses → OpenClaw Shield: may include
- Security defenses → SecureClaw: may include
- Security defenses → ClawKeeper: may include