How sudarshan2412/taskverifier works
TaskVerifier generates C proof-of-concept exploits for vulnerability tasks, then iteratively compiles and runs them, returning structured feedback to the LLM until a crash is reproduced or attempts are exhausted. Its main path is dataset selection and sanitization → agent generation/refinement → verifier checks → result and pass/fail evaluation. The README also documents one-shot baseline runs and batch comparison/metrics. Some subsystem links reflect README documentation because those source files were not sampled.
Main entry points: agent_loop.py Retry loop · __init__.py Verifier pipeline · run_pipeline.py Trial pipeline · results.json Trial results
Components 24
Task orchestration
- CVE subset
cybergym_subset.json - Trial pipeline
run_pipeline.py - Entry sanitizer
dataset_sanitizer.py
PoC generation
- Retry loop
agent/agent_loop.py - Prompt builder
agent/prompt_builder.py - Conversation context
agent/context_manager.py - LLM client
agent/llm_client.py - Code extractor
agent/code_extractor.py - Format hints
agent/format_hints.py
Verification
- Verifier pipeline
verifier/__init__.py - Compiler checks
verifier/compiler.py - Sandbox execution
verifier/execution.py - Crash parsing
verifier/sanitizer.py - Feedback builder
verifier/feedback_builder.py - Symbol detection
verifier/hallucination_detector.py
Benchmark and evaluation
- Pass/fail evaluator
evaluator.py - Baseline runner
baseline_runner.py - Trial runner
runner.py - Metrics analysis
compute_metrics.py - Result plots
plot_results.py - Trial results
results.json
Other components
- Researcher
- OpenRouter API
- Vulnerable target
Connections 25
- Researcher → Trial pipeline: starts trials
- CVE subset → Trial pipeline: supplies tasks
- Trial pipeline → Entry sanitizer: sanitizes entries
- Trial pipeline → Retry loop: runs trials
- Retry loop → Prompt builder: builds prompts
- Retry loop → Conversation context: manages history
- Retry loop → LLM client: requests generation
- LLM client → OpenRouter API: sends requests
- Retry loop → Code extractor: extracts code
- Prompt builder → Format hints: uses format hints
- Retry loop → Symbol detection: checks symbols
- Retry loop → Verifier pipeline: submits PoC
- Verifier pipeline → Compiler checks: compiles PoC
- Verifier pipeline → Sandbox execution: runs compiled PoC
- Sandbox execution → Vulnerable target: runs in sandbox
- Verifier pipeline → Crash parsing: parses crash output
- Verifier pipeline → Feedback builder: builds guidance
- Verifier pipeline → Retry loop: returns feedback
- Trial pipeline → Pass/fail evaluator: evaluates outcome
- Baseline runner → Trial runner: runs baseline trials
- Trial runner → Retry loop: invokes agent
- Trial runner → Trial results: records outcomes
- Trial pipeline → Trial results: writes outcomes
- Trial results → Metrics analysis: aggregates results
- Trial results → Result plots: visualizes results