How bhaskararamkarri/debugarena works
DebugArena evaluates coding agents on buggy repository tasks: the runner loads a task into the environment, an agent repeatedly proposes JSON edits, commands, or submission, and the environment uses a sandbox and hidden tests to score progress. Completed episodes are judged and logged for review, dashboard analysis, and SFT/DPO dataset export. The core loop and runner ownership are visible in excerpts; dashboard and export connections are documented in the README, while the export implementation itself was not sampled.
Main entry points: runner.py Episode runner · env.py Bug-fix environment · runs Episode trajectories · agent.py Coding agent
Components 15
Task Benchmark
- Task definitions
tasks
Agent Evaluation
- Episode runner
agentgym/runner.py - Bug-fix environment
agentgym/env.py - Coding agent
agentgym/agent.py - Isolated execution
agentgym/sandbox.py - Reward scoring
agentgym/reward.py - Fix quality judge
agentgym/judge.py - Episode tracing
agentgym/tracer.py
Evaluation Artifacts
- Episode trajectories
runs - Run summaries
runs
Review and Training
- Results dashboard
dashboard/app.py - SFT / DPO datasets
dataset
Other components
- Evaluation operator
- LLM provider
- LangSmith
Connections 15
- Evaluation operator → Episode runner: starts evaluation
- Episode runner → Task definitions: loads task data
- Episode runner → Bug-fix environment: resets and steps
agentgym/runner.py - Episode runner → Coding agent: requests actions
agentgym/runner.py - Coding agent → LLM provider: requests completions
- Bug-fix environment → Isolated execution: writes and executes
agentgym/env.py - Bug-fix environment → Reward scoring: computes reward
agentgym/env.py - Bug-fix environment → Isolated execution: runs hidden tests
agentgym/env.py - Episode runner → Fix quality judge: evaluates final fix
agentgym/runner.py - Episode runner → Episode trajectories: writes episode steps
agentgym/runner.py - Episode runner → Run summaries: writes run results
agentgym/runner.py - Episode runner → Episode tracing: records trace spans
agentgym/runner.py - Episode tracing → LangSmith: sends spans
- Episode trajectories → Results dashboard: displays run data
README.md - Episode trajectories → SFT / DPO datasets: exports training data
README.md