How yuyangsunshine/bioprobench works
Purpose and Entry Points
BioProBench is a data-and-evaluation benchmark for LLM understanding, correction, ordering, generation, and reasoning over biological protocols. It is not an application service; the main execution paths are batch inference followed by task-specific scoring.
Primary entry points:
• Remote/API model inference: Scripts/generate_response.py
• Local Hugging Face model inference: Scripts/generate_response_local.py
• Legacy standalone metrics: Metrics/PQA.py, Metrics/ORD.py, Metrics/ERR.py, Metrics/GEN.py, Metrics/REA-ERR.py
• Portable evaluation package: skills/evaluate-protocol-outputs/scripts/evaluate.py
Benchmark Data Layer
Task examples are JSON datasets, with representative test splits committed under Data/: Data/PQA_test.json, Data/ORD_test.json, Data/ERR_test.json, and Data/GEN_test.json. The README describes larger train sets and source corpora as externally distributed through Hugging Face.
The task contract is central: PQA answers protocol questions; ORD reconstructs step sequence; ERR detects/corrects errors; GEN generates protocols; REA covers reasoning variants. Input/output conventions for the portable evaluator are documented in skills/evaluate-protocol-outputs/references/input-formats.md.
Inference and Response Production
Inference scripts select a task and dataset, format prompts, invoke a configured model, and persist model responses for later scoring:
• Scripts/generate_response.py targets OpenAI-compatible APIs and can be configured for providers/models.
• Scripts/generate_response_local.py runs local Hugging Face models.
• Scripts/prompt_format.py is the shared prompt-construction boundary.
External dependencies are LLM API endpoints for remote execution and Hugging Face model artifacts/runtime for local execution. API keys, base URLs, model names, and task names are script-level configuration rather than a centralized service configuration.
Legacy Metric Pipelines
The original evaluator is split by benchmark task and expects a response-file path configured in each script:
• QA: Metrics/PQA.py
• Ordering: Metrics/ORD.py
• Error correction: Metrics/ERR.py
• Generation: Metrics/GEN.py
• Reasoning/error consistency: Metrics/REA-ERR.py
These scripts transform prediction/reference records into task metrics such as accuracy/Brier score, exact match/Kendall tau, precision/recall/F1, BLEU, and step recall/precision. They form a file-based boundary: inference output JSON is consumed offline, without orchestration or persistent backend.
Portable Agent Skill
skills/evaluate-protocol-outputs/ packages the evaluation logic for independent use by agent frameworks. Its contract and scope are in skills/evaluate-protocol-outputs/SKILL.md; agent registration metadata is in skills/evaluate-protocol-outputs/agents/openai.yaml.
The portable orchestration entry point is skills/evaluate-protocol-outputs/scripts/evaluate.py, which routes normalized inputs to task-specific evaluators rather than relying on the older Metrics/ scripts.
Evaluation Stages and Semantic Boundaries
The portable pipeline is organized as:
1. Normalize heterogeneous prediction/reference payloads: skills/evaluate-protocol-outputs/scripts/normalize.py.
2. Dispatch by task in skills/evaluate-protocol-outputs/scripts/evaluate.py.
3. Apply task semantics: generation.py, qa.py, ordering.py, and error_detection.py.
4. Assess judge-based reasoning consistency when applicable: skills/evaluate-protocol-outputs/scripts/judge_consistency.py.
Metric definitions and benchmark-compatible semantics are specified in skills/evaluate-protocol-outputs/references/metric-definitions.md and skills/evaluate-protocol-outputs/references/bioprobench-profile.md. This separates canonical parsing/scoring behavior from model invocation and keeps the skill usable in external agent runtimes.
Key Architectural Constraint
The repository has two parallel evaluation surfaces: standalone research scripts in Metrics/ and the newer self-contained skill. New integrations should target skills/evaluate-protocol-outputs/scripts/evaluate.py and its normalization layer; retain Metrics/ primarily for reproduction compatibility. Both depend on correctly structured JSON responses, making output schema adherence the critical integration boundary.
Main entry points: generate_response.py Remote inference · generate_response_local.py Local inference · evaluate.py Evaluation dispatcher · prompt_format.py Prompt construction
Components 19
Benchmark data
- Task test datasets · PQA, ORD, ERR, and GEN test splits.
Data/PQA_test.json - Task contract · Defines PQA, ORD, ERR, GEN, and REA conventions.
skills/evaluate-protocol-outputs/references/input-formats.md
Inference
- Prompt construction
Scripts/prompt_format.py - Remote inference · Runs configured OpenAI-compatible providers.
Scripts/generate_response.py - Local inference · Runs local Hugging Face models.
Scripts/generate_response_local.py - LLM API endpoints
- HF models and runtime
- Response JSON · Schema adherence is the shared scoring boundary.
Legacy metrics
Retained for reproduction compatibility.
- QA metrics · Accuracy and Brier-style scoring.
Metrics/PQA.py - Ordering metrics · Exact match and Kendall tau.
Metrics/ORD.py - Error metrics · Precision, recall, and F1 scoring.
Metrics/ERR.py - Generation metrics · BLEU and step recall/precision.
Metrics/GEN.py - Reasoning consistency
Metrics/REA-ERR.py
Portable evaluation skill
Preferred integration surface.
- Skill contract · Portable evaluator scope and agent registration.
skills/evaluate-protocol-outputs/SKILL.md - Normalize payloads · Canonicalizes heterogeneous prediction and reference payloads.
skills/evaluate-protocol-outputs/scripts/normalize.py - Evaluation dispatcher · Routes normalized inputs by task.
skills/evaluate-protocol-outputs/scripts/evaluate.py - Task evaluators · Generation, QA, ordering, and error-detection semantics.
skills/evaluate-protocol-outputs/scripts/generation.py - Judge consistency · Optional judge-based reasoning consistency assessment.
skills/evaluate-protocol-outputs/scripts/judge_consistency.py - Metric profile · Benchmark-compatible metric definitions and semantics.
skills/evaluate-protocol-outputs/references/metric-definitions.md
Connections 21
- Task test datasets → Remote inference: task inputs
- Task test datasets → Local inference: task inputs
- Task contract → Prompt construction: formats prompts
- Prompt construction → Remote inference: prompts
- Prompt construction → Local inference: prompts
- Remote inference → LLM API endpoints: invokes
- Local inference → HF models and runtime: loads
- Remote inference → Response JSON: writes
- Local inference → Response JSON: writes
- Response JSON → QA metrics: offline scoring
- Response JSON → Ordering metrics: offline scoring
- Response JSON → Error metrics: offline scoring
- Response JSON → Generation metrics: offline scoring
- Response JSON → Reasoning consistency: offline scoring
- Response JSON → Normalize payloads: predictions and references
- Normalize payloads → Evaluation dispatcher: normalized records
- Evaluation dispatcher → Task evaluators: dispatches by task
- Evaluation dispatcher → Judge consistency: when applicable
- Metric profile → Task evaluators: defines semantics
- Skill contract → Evaluation dispatcher: packages
- Task contract → Normalize payloads: input conventions