How sharathbaddam/agentwhetters-cybergym works
1. What this repo is
- This is a Python-based “purple agent” for CyberGym/AgentBeats that generates exploit PoCs against a vulnerable program provided by a separate green-agent benchmark harness.
- The main runtime is a long-lived agent service; the repo also includes a client/runner package for driving benchmark scenarios.
2. Top-level split: service side vs. client side
- src/ contains the production agent server and reasoning pipeline.
- agentbeats/ contains client-facing helpers, CLI, and models for interacting with the AgentBeats/CyberGym task flow.
- scenario*.toml files and run_scenario.py indicate scripted benchmark runs rather than a general-purpose application UI.
3. Inbound task boundary
- The green agent sends a task bundle to the server over an A2A-style interface: task text plus attachments such as description.txt, error.txt, and a vulnerable source tarball.
- src/server.py exposes the server on port 9122.
- src/executor.py keeps one Agent instance per context_id, so each benchmark task has isolated state and concurrent tasks do not interfere.
4. Core control loop
- src/agent.py is the orchestration center.
- It runs the entire exploit-generation loop: pre-analysis, LLM prompting, tool execution, PoC submission, feedback handling, retry, and final success reporting.
- The agent caps work with both iteration and token budgets, so the architecture is explicitly cost-governed rather than open-ended.
5. Zero-token pre-analysis pipeline
- Before any model call, the agent runs structured analysis modules that do not consume LLM tokens:
- hypothesis_parser.py extracts a normalized vulnerability signal from the task description and error text.
- codebase_triage.py ranks source files in the vulnerable repo by likely relevance to the detected vulnerability class.
- taint_extractor.py traces likely input-to-crash paths and identifies trigger fields, magic bytes, and format structure.
- memory_system.py retrieves similar past tasks and known failed strategies.
- This is the main architectural optimization: the LLM is only called after the search space has been narrowed.
6. LLM interaction boundary
- The agent supports two OpenAI API modes: Responses API for newer reasoning models and Chat Completions for classic models.
- The LLM is not asked to discover everything from scratch; it receives parsed vulnerability metadata, ranked files, taint/path hints, and warm-start context.
- The model can invoke two tools:
- execute_python to synthesize or transform PoC bytes locally.
- submit_poc to send a candidate artifact back through the benchmark round-trip for testing.
7. Feedback loop and repair
- After the green agent tests a PoC, feedback_classifier.py reduces raw exits/ASan output into a small set of categories such as NO_CRASH, WRONG_LOCATION, PARTIAL_CRASH, or PARSER_REJECTED.
- binary_mutator.py then handles near-miss cases by producing deterministic byte-level mutations without another LLM call.
- Only when the classifier says the issue needs reasoning-level adjustment does the agent spend another model iteration.
8. Cross-task learning and persistence
- memory_system.py stores successful exploit metadata and prior failed strategies in JSON-backed memory, so later tasks can reuse patterns from similar vulnerability classes or domains.
- token_tracker.py records model usage, latency, and estimated cost and enforces per-task budgets.
- These are persistence boundaries: the exploit loop is ephemeral, but memory and token accounting survive across tasks for analysis and reuse.
9. Support code and observability
- log_config.py defines a custom TRACE level for deep debugging of the agent’s internal decisions.
- src/messenger.py and src/run_scenario.py support task messaging and scenario execution around the main agent flow.
- The repo includes a Dockerfile for containerized execution and .github/workflows/test-and-publish.yml for CI/release automation.
10. Testing and validation
- tests/ covers parser behavior, feedback classification, agent behavior, and integration paths.
- The test surface reflects the architecture: most correctness risk is in structured extraction, feedback interpretation, and end-to-end task handling rather than in a traditional web backend.
11. Important external dependencies and runtime assumptions
- The agent depends on Python, uv, and either OpenAI or Azure OpenAI for the reasoning loop.
- It also assumes CyberGym’s green-agent test harness and A2A-style task exchange.
- taint_extractor.py can use cscope when present, but falls back to grep-style analysis, so source-code tracing is designed to degrade gracefully.
12. How to diagram it
- The clearest architecture diagram is: green-agent task ingress → per-context executor → agent orchestration → zero-token analysis modules + memory retrieval → LLM/tool loop → green-agent test round-trip → classifier/mutator repair loop → success artifact and memory persistence.
- The main system boundary is the agent service; the benchmark harness and client package are external consumers, not part of the core exploit reasoning pipeline.
Main entry points: agent.py Agent loop · client.py Client · server.py Server · executor.py Executor
Components 21
Agent service
Core purple-agent runtime and reasoning pipeline
- Server · Inbound task service exposed on port 9122
src/server.py - Executor · Maintains one agent instance per context_id
src/executor.py - Agent loop · Main exploit-generation control loop with budgets and retries
src/agent.py - Hypothesis parser · Normalizes vulnerability signals from task text and errors
src/hypothesis_parser.py - Code triage · Ranks source files by likely relevance
src/codebase_triage.py - Taint extractor · Finds likely input-to-crash paths and trigger structure
src/taint_extractor.py - Feedback classifier · Maps test output to coarse feedback categories
src/feedback_classifier.py - Mutator · Performs deterministic near-miss PoC mutations
src/binary_mutator.py - Messenger · Supports task exchange and scenario messaging
src/messenger.py
Client side
Benchmark-driving helpers and scenario entrypoints
- Client · Client-facing task runner and benchmark helper
agentbeats/client.py - CLI · Command-line entrypoint for running tasks
agentbeats/client_cli.py - Scenario runner · Runs scripted benchmark scenarios
agentbeats/run_scenario.py - Models · Shared data models for AgentBeats flow
agentbeats/models.py
State
Cross-task memory and accounting data
- Memory · Retrieves and stores similar tasks and strategies
src/memory_system.py - Token tracker · Tracks usage, latency, and per-task limits
src/token_tracker.py - Memory store · Persisted solved tasks and token usage records
src/memory/solved_tasks.json - Usage store · Persisted token usage history
src/memory/token_usage.json
Ops
Containerization and release automation
- Dockerfile · Containerized execution environment
Dockerfile - CI workflow · Test and publish automation
.github/workflows/test-and-publish.yml
Other components
- Green harness · CyberGym/AgentBeats task source and PoC tester
- LLM API · OpenAI or Azure OpenAI backend used by the agent
Connections 22
- Green harness → Server: task bundle · Green-agent task ingress with attachments
- Server → Executor: dispatch · Routes incoming tasks by context_id
- Executor → Agent loop: context agent · Creates or reuses one agent per task context
- Agent loop → Hypothesis parser: pre-analyze · Extracts normalized vulnerability signals
- Agent loop → Code triage: pre-analyze · Ranks relevant source files
- Agent loop → Taint extractor: pre-analyze · Derives trigger paths and field hints
- Agent loop → Memory: retrieve · Loads similar tasks and prior failures
- Memory → Memory store: persist · Reads and writes JSON-backed task memory
- Agent loop → Token tracker: budget · Enforces token and latency limits
- Token tracker → Usage store: persist · Stores usage accounting
- Agent loop → LLM API: reason · Calls Responses API or Chat Completions
- Agent loop → Feedback classifier: analyze result · Reduces test output to coarse categories
- Feedback classifier → Mutator: near miss · Triggers deterministic repair mutations
- Mutator → Agent loop: retry · Feeds improved candidate back into the loop
- Agent loop → Messenger: submit · Sends PoCs and receives benchmark feedback
- CLI → Client: invoke · CLI drives client helpers
- Scenario runner → Client: run · Scenario execution uses the client package
- Client → Green harness: drive tasks · Client participates in benchmark task flow
- Client → Server: agent task · Submits or coordinates task exchange with the service
- Models → Client: types · Shared request and response structures
- Dockerfile → Server: package · Containerizes the runtime
- CI workflow → Dockerfile: build test · CI validates and publishes containerized release