How ryandegl233/autoempirical works
AutoEmpirical is a dataset-first benchmark for testing automated empirical software-fault analysis. Its central workflow moves paper-specific research objects through collection, relevance filtering, and taxonomy labeling (Stages 1–3). Around that dataset, the repository provides provenance and source evidence, baseline and adaptive analysis implementations, frozen experiment inputs, and external evidence-capture tools. The README documents these capabilities; the sampled source mainly shows experiment transport and continuation safeguards, so runtime wiring beyond those excerpts is represented only where documented.
Main entry points: adaptive_empirical_workflow Adaptive Workflow · scripts Experiment Runner · stage1.csv Candidate Collection · stage2.csv Relevant Bugs
Components 20
Research Dataset
- Candidate Collection
Dataset/stage1.csv - Relevant Bugs
Dataset/stage2.csv - Taxonomy Labels
Dataset/stage3.csv - Per-Paper Data
Dataset/by_paper
Evidence And Provenance
- Source Evidence
Dataset/evidence - Information Reconstruction
Dataset/reconstruction - Provenance Audits
Dataset/scripts
Analysis Workflows
- Single-LLM Baseline
Benchmark/src - CAMEL Baseline
Benchmark/src - Adaptive Workflow
Benchmark/src/adaptive_empirical_workflow - Independent Analysis
Benchmark/src/adaptive_empirical_workflow - Evidence Verification
Benchmark/src/adaptive_empirical_workflow - Arbitration
Benchmark/src/adaptive_empirical_workflow - Execution Audits
Benchmark/src/adaptive_empirical_workflow
Benchmark Experiments
- Frozen Experiment Inputs
Benchmark/inputs - Experiment Configurations
Benchmark/configs - Experiment Runner
Benchmark/scripts
Other components
- Researcher
- External Sources
- LLM Provider
Connections 19
- Researcher → Experiment Runner: runs experiments
- Candidate Collection → Relevant Bugs: filters candidates
- Relevant Bugs → Taxonomy Labels: labels taxonomy
- Per-Paper Data → Candidate Collection: unifies records
- Source Evidence → Provenance Audits: audits evidence
- Information Reconstruction → Per-Paper Data: integrates findings
- External Sources → Source Evidence: supplies evidence
- Frozen Experiment Inputs → Experiment Runner: provides cohorts
- Experiment Configurations → Experiment Runner: configures runs
- Experiment Runner → Single-LLM Baseline: invokes baseline
- Experiment Runner → CAMEL Baseline: invokes baseline
- Experiment Runner → Adaptive Workflow: invokes workflow
- Adaptive Workflow → Independent Analysis: coordinates analysis
- Adaptive Workflow → Evidence Verification: coordinates verification
- Adaptive Workflow → Arbitration: coordinates arbitration
- Adaptive Workflow → Execution Audits: records audits
- Single-LLM Baseline → LLM Provider: sends requests
- CAMEL Baseline → LLM Provider: sends requests
- Adaptive Workflow → LLM Provider: sends requests