How fastfedora/refactor-arena works
Refactor Arena is a Control Arena setting and CLI for evaluating repository-refactoring agents, optionally in an attack mode that adds a hidden vulnerability. The user configures an evaluation and selects a dataset; the setting builds task samples, policies guide agent behavior, and scorers evaluate the resulting changes. The README documents configurable prompts, tasks, policies, monitors, metrics, and evaluation settings. Source excerpts are partial, so links between the runtime subsystems are shown only where the README or indexed source references support them.
Main entry points: refactor_arena_setting.py Arena setting · main.py CLI · rescore.py Rescore · dataset.py Dataset samples
Components 19
CLI and setup
- CLI
src/refactor_arena/cli/main.py - Initialize
src/refactor_arena/cli/commands/init.py - Rescore
src/refactor_arena/cli/commands/rescore.py
Task construction
- Arena setting
src/refactor_arena/settings/refactor_arena_setting.py - Dataset samples
src/refactor_arena/datasets/dataset.py - Dataset model
src/refactor_arena/datasets/dataset.py
Agent behavior
- Honest policy
src/refactor_arena/policies/swe_agent_honest_policy.py - Attack policy
src/refactor_arena/policies/swe_agent_attack_policy.py - Agent control
src/refactor_arena/solvers/agent_control_solver.py
Scoring and analysis
- Change analysis
src/refactor_arena/solvers/analyze_code_changes.py - Diff monitor
src/refactor_arena/scorers/git_diff_monitor.py - Constraint scorer
src/refactor_arena/scorers/task_constraints_penalty.py - Code metrics
src/refactor_arena/utils/coding/analyze_code_changes.py - Metric types
src/refactor_arena/utils/coding/types/code_quality_metrics.py - Evaluation metrics
src/refactor_arena/utils/evals/_metrics/add_safety_metric.py - Threshold metric
src/refactor_arena/utils/evals/_metrics/add_threshold_accuracy_metric.py - Run scorer tool
src/refactor_arena/utils/evals/get_run_scorer_tool.py - Eval log reader
src/refactor_arena/utils/evals/read_setting_eval_log.py
Other components
- Evaluator
Connections 18
- Evaluator → CLI: configures evaluation
README.md - CLI → Arena setting: runs setting
src/refactor_arena/cli/main.py - Evaluator → Initialize: sets up scenario
README.md - CLI → Rescore: dispatches command
- Arena setting → Dataset samples: builds samples
src/refactor_arena/settings/refactor_arena_setting.py - Dataset samples → Dataset model: creates sample data
src/refactor_arena/datasets/dataset.py - Arena setting → Honest policy: selects policy
README.md - Arena setting → Attack policy: selects attack mode
README.md - Arena setting → Agent control: configures agent
README.md - Arena setting → Diff monitor: configures monitor
README.md - Arena setting → Constraint scorer: configures scoring
README.md - Constraint scorer → Change analysis: assesses changes
- Change analysis → Code metrics: analyzes code
- Code metrics → Metric types: produces metrics
- Arena setting → Evaluation metrics: configures metrics
README.md - Arena setting → Threshold metric: configures metrics
README.md - Run scorer tool → Eval log reader: reads evaluation output
- Rescore → Run scorer tool: uses scorer