How mathming/hks930 works
ARC-Bench Agent is an Octos-based adapter that turns benchmark requirements into a generated web application and checks requirement acceptance. The main flow is documented as runner input through the Octos pipeline, with per-requirement verification; local tools also provide setup, grading, and coverage auditing. The README and excerpts establish the adapter/runtime, policy, prompts, and starter-template roles. Source excerpts are partial, so unshown pipeline internals and connections are omitted.
Main entry points: main.py ARC Adapter · run-task-local.py Local Task Runner · verify_node.py Requirement Verifier · grade-local.py Local Grader
Components 15
Agent Execution
- ARC Adapter
arc/main.py - Execution Policy
arc/arc-policy.toml - Agent Prompts
arc/prompts - Octos Runtime
- Octos Protocol
arc/octos_stdio.py - Web Starter
arc/template
Acceptance Checks
- Requirement Verifier
arc/verify_node.py - Browser Specs
arc/public-tests
Local Evaluation
- Local Task Runner
arc/run-task-local.py - API Settings
arc/local_api_config.py - Runtime Environment
arc/local_runtime.py - Local Grader
arc/grade-local.py - Coverage Audit
arc/github_eval.py
Other components
- Benchmark Runner
- Generated Web App
Connections 15
- Benchmark Runner → ARC Adapter: provides requirements
README.md - ARC Adapter → Octos Runtime: runs pipeline
README.md - ARC Adapter → Execution Policy: uses policy
README.md - ARC Adapter → Agent Prompts: uses prompts
README.md - ARC Adapter → Octos Protocol: connects through
arc/main.py - ARC Adapter → Web Starter: uses starter
README.md - Octos Runtime → Generated Web App: generates
README.md - ARC Adapter → Requirement Verifier: checks acceptance
README.md - Requirement Verifier → Generated Web App: verifies
arc/verify_node.py - Requirement Verifier → Browser Specs: runs specs
- Local Task Runner → ARC Adapter: launches
arc/run-task-local.py - Local Task Runner → API Settings: resolves settings
arc/run-task-local.py - Local Task Runner → Runtime Environment: sets environment
arc/run-task-local.py - Local Grader → Generated Web App: grades
- Local Grader → Coverage Audit: audits coverage
arc/grade-local.py