# yuyangsunshine/bioprobench architecture

> Architecture diagram of the GitHub repository yuyangsunshine/bioprobench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yuyangsunshine/bioprobench
- Repository: https://github.com/yuyangsunshine/bioprobench

## Overview

#### Purpose and Entry Points
BioProBench is a data-and-evaluation benchmark for LLM understanding, correction, ordering, generation, and reasoning over biological protocols. It is not an application service; the main execution paths are batch inference followed by task-specific scoring.

Primary entry points:
- Remote/API model inference: `Scripts/generate_response.py`
- Local Hugging Face model inference: `Scripts/generate_response_local.py`
- Legacy standalone metrics: `Metrics/PQA.py`, `Metrics/ORD.py`, `Metrics/ERR.py`, `Metrics/GEN.py`, `Metrics/REA-ERR.py`
- Portable evaluation package: `skills/evaluate-protocol-outputs/scripts/evaluate.py`

#### Benchmark Data Layer
Task examples are JSON datasets, with representative test splits committed under `Data/`: `Data/PQA_test.json`, `Data/ORD_test.json`, `Data/ERR_test.json`, and `Data/GEN_test.json`. The README describes larger train sets and source corpora as externally distributed through Hugging Face.

The task contract is central: PQA answers protocol questions; ORD reconstructs step sequence; ERR detects/corrects errors; GEN generates protocols; REA covers reasoning variants. Input/output conventions for the portable evaluator are documented in `skills/evaluate-protocol-outputs/references/input-formats.md`.

#### Inference and Response Production
Inference scripts select a task and dataset, format prompts, invoke a configured model, and persist model responses for later scoring:
- `Scripts/generate_response.py` targets OpenAI-compatible APIs and can be configured for providers/models.
- `Scripts/generate_response_local.py` runs local Hugging Face models.
- `Scripts/prompt_format.py` is the shared prompt-construction boundary.

External dependencies are LLM API endpoints for remote execution and Hugging Face model artifacts/runtime for local execution. API keys, base URLs, model names, and task names are script-level configuration rather than a centralized service configuration.

#### Legacy Metric Pipelines
The original evaluator is split by benchmark task and expects a response-file path configured in each script:
- QA: `Metrics/PQA.py`
- Ordering: `Metrics/ORD.py`
- Error correction: `Metrics/ERR.py`
- Generation: `Metrics/GEN.py`
- Reasoning/error consistency: `Metrics/REA-ERR.py`

These scripts transform prediction/reference records into task metrics such as accuracy/Brier score, exact match/Kendall tau, precision/recall/F1, BLEU, and step recall/precision. They form a file-based boundary: inference output JSON is consumed offline, without orchestration or persistent backend.

#### Portable Agent Skill
`skills/evaluate-protocol-outputs/` packages the evaluation logic for independent use by agent frameworks. Its contract and scope are in `skills/evaluate-protocol-outputs/SKILL.md`; agent registration metadata is in `skills/evaluate-protocol-outputs/agents/openai.yaml`.

The portable orchestration entry point is `skills/evaluate-protocol-outputs/scripts/evaluate.py`, which routes normalized inputs to task-specific evaluators rather than relying on the older `Metrics/` scripts.

#### Evaluation Stages and Semantic Boundaries
The portable pipeline is organized as:
1. Normalize heterogeneous prediction/reference payloads: `skills/evaluate-protocol-outputs/scripts/normalize.py`.
2. Dispatch by task in `skills/evaluate-protocol-outputs/scripts/evaluate.py`.
3. Apply task semantics: `generation.py`, `qa.py`, `ordering.py`, and `error_detection.py`.
4. Assess judge-based reasoning consistency when applicable: `skills/evaluate-protocol-outputs/scripts/judge_consistency.py`.

Metric definitions and benchmark-compatible semantics are specified in `skills/evaluate-protocol-outputs/references/metric-definitions.md` and `skills/evaluate-protocol-outputs/references/bioprobench-profile.md`. This separates canonical parsing/scoring behavior from model invocation and keeps the skill usable in external agent runtimes.

#### Key Architectural Constraint
The repository has two parallel evaluation surfaces: standalone research scripts in `Metrics/` and the newer self-contained skill. New integrations should target `skills/evaluate-protocol-outputs/scripts/evaluate.py` and its normalization layer; retain `Metrics/` primarily for reproduction compatibility. Both depend on correctly structured JSON responses, making output schema adherence the critical integration boundary.

Main entry points: [`Scripts/generate_response.py`](https://github.com/yuyangsunshine/bioprobench/blob/main/Scripts/generate_response.py) (Remote inference), [`Scripts/generate_response_local.py`](https://github.com/yuyangsunshine/bioprobench/blob/main/Scripts/generate_response_local.py) (Local inference), [`skills/evaluate-protocol-outputs/scripts/evaluate.py`](https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/scripts/evaluate.py) (Evaluation dispatcher), [`Scripts/prompt_format.py`](https://github.com/yuyangsunshine/bioprobench/blob/main/Scripts/prompt_format.py) (Prompt construction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Benchmark data"]
  node_datasets["Task test datasets<br/>JSON datasets<br/>[PQA_test.json]"]
  node_task_contract["Task contract<br/>benchmark semantics<br/>[input-formats.md]"]
end

subgraph group_inference["Inference"]
  node_prompt_format["Prompt construction<br/>shared formatter<br/>[prompt_format.py]"]
  node_remote_inference["Remote inference<br/>API runner"]
  node_local_inference["Local inference<br/>HF runner"]
  node_llm_apis{{"LLM API endpoints<br/>external service"}}
  node_hf_runtime{{"HF models and runtime<br/>external artifacts"}}
  node_response_json["Response JSON<br/>file contract"]
end

subgraph group_legacy["Legacy metrics"]
  node_legacy_qa["QA metrics<br/>legacy evaluator<br/>[PQA.py]"]
  node_legacy_ordering["Ordering metrics<br/>legacy evaluator<br/>[ORD.py]"]
  node_legacy_error["Error metrics<br/>legacy evaluator<br/>[ERR.py]"]
  node_legacy_generation["Generation metrics<br/>legacy evaluator<br/>[GEN.py]"]
  node_legacy_reasoning["Reasoning consistency<br/>legacy evaluator<br/>[REA-ERR.py]"]
end

subgraph group_skill["Portable evaluation skill"]
  node_skill_contract["Skill contract<br/>agent package<br/>[SKILL.md]"]
  node_normalize["Normalize payloads<br/>normalization layer<br/>[normalize.py]"]
  node_evaluate["Evaluation dispatcher<br/>skill entry point<br/>[evaluate.py]"]
  node_task_evaluators["Task evaluators<br/>semantic scoring<br/>[generation.py]"]
  node_judge_consistency["Judge consistency<br/>reasoning evaluator"]
  node_metric_profile["Metric profile<br/>evaluation specification"]
end

node_datasets -->|"task inputs"| node_remote_inference
node_datasets -->|"task inputs"| node_local_inference
node_task_contract -->|"formats prompts"| node_prompt_format
node_prompt_format -->|"prompts"| node_remote_inference
node_prompt_format -->|"prompts"| node_local_inference
node_remote_inference -->|"invokes"| node_llm_apis
node_local_inference -->|"loads"| node_hf_runtime
node_remote_inference -->|"writes"| node_response_json
node_local_inference -->|"writes"| node_response_json
node_response_json -->|"offline scoring"| node_legacy_qa
node_response_json -->|"offline scoring"| node_legacy_ordering
node_response_json -->|"offline scoring"| node_legacy_error
node_response_json -->|"offline scoring"| node_legacy_generation
node_response_json -->|"offline scoring"| node_legacy_reasoning
node_response_json -->|"predictions and references"| node_normalize
node_normalize -->|"normalized records"| node_evaluate
node_evaluate -->|"dispatches by task"| node_task_evaluators
node_evaluate -.->|"when applicable"| node_judge_consistency
node_metric_profile -->|"defines semantics"| node_task_evaluators
node_skill_contract -->|"packages"| node_evaluate
node_task_contract -->|"input conventions"| node_normalize

click node_datasets "https://github.com/yuyangsunshine/bioprobench/blob/main/Data/PQA_test.json"
click node_task_contract "https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/references/input-formats.md"
click node_prompt_format "https://github.com/yuyangsunshine/bioprobench/blob/main/Scripts/prompt_format.py"
click node_remote_inference "https://github.com/yuyangsunshine/bioprobench/blob/main/Scripts/generate_response.py"
click node_local_inference "https://github.com/yuyangsunshine/bioprobench/blob/main/Scripts/generate_response_local.py"
click node_legacy_qa "https://github.com/yuyangsunshine/bioprobench/blob/main/Metrics/PQA.py"
click node_legacy_ordering "https://github.com/yuyangsunshine/bioprobench/blob/main/Metrics/ORD.py"
click node_legacy_error "https://github.com/yuyangsunshine/bioprobench/blob/main/Metrics/ERR.py"
click node_legacy_generation "https://github.com/yuyangsunshine/bioprobench/blob/main/Metrics/GEN.py"
click node_legacy_reasoning "https://github.com/yuyangsunshine/bioprobench/blob/main/Metrics/REA-ERR.py"
click node_skill_contract "https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/SKILL.md"
click node_normalize "https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/scripts/normalize.py"
click node_evaluate "https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/scripts/evaluate.py"
click node_task_evaluators "https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/scripts/generation.py"
click node_judge_consistency "https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/scripts/judge_consistency.py"
click node_metric_profile "https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/references/metric-definitions.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_datasets,node_task_contract toneBlue
class node_prompt_format,node_remote_inference,node_local_inference,node_llm_apis,node_hf_runtime,node_response_json toneAmber
class node_legacy_qa,node_legacy_ordering,node_legacy_error,node_legacy_generation,node_legacy_reasoning toneMint
class node_skill_contract,node_normalize,node_evaluate,node_task_evaluators,node_judge_consistency,node_metric_profile toneRose
```

## Components

### Benchmark data

- [Task test datasets](https://github.com/yuyangsunshine/bioprobench/blob/main/Data/PQA_test.json): JSON datasets · PQA, ORD, ERR, and GEN test splits. · `Data/PQA_test.json`
- [Task contract](https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/references/input-formats.md): benchmark semantics · Defines PQA, ORD, ERR, GEN, and REA conventions. · `skills/evaluate-protocol-outputs/references/input-formats.md`

### Inference

- [Prompt construction](https://github.com/yuyangsunshine/bioprobench/blob/main/Scripts/prompt_format.py): shared formatter · `Scripts/prompt_format.py`
- [Remote inference](https://github.com/yuyangsunshine/bioprobench/blob/main/Scripts/generate_response.py): API runner · Runs configured OpenAI-compatible providers. · `Scripts/generate_response.py`
- [Local inference](https://github.com/yuyangsunshine/bioprobench/blob/main/Scripts/generate_response_local.py): HF runner · Runs local Hugging Face models. · `Scripts/generate_response_local.py`
- **LLM API endpoints**: external service
- **HF models and runtime**: external artifacts
- **Response JSON**: file contract · Schema adherence is the shared scoring boundary.

### Legacy metrics

Retained for reproduction compatibility.

- [QA metrics](https://github.com/yuyangsunshine/bioprobench/blob/main/Metrics/PQA.py): legacy evaluator · Accuracy and Brier-style scoring. · `Metrics/PQA.py`
- [Ordering metrics](https://github.com/yuyangsunshine/bioprobench/blob/main/Metrics/ORD.py): legacy evaluator · Exact match and Kendall tau. · `Metrics/ORD.py`
- [Error metrics](https://github.com/yuyangsunshine/bioprobench/blob/main/Metrics/ERR.py): legacy evaluator · Precision, recall, and F1 scoring. · `Metrics/ERR.py`
- [Generation metrics](https://github.com/yuyangsunshine/bioprobench/blob/main/Metrics/GEN.py): legacy evaluator · BLEU and step recall/precision. · `Metrics/GEN.py`
- [Reasoning consistency](https://github.com/yuyangsunshine/bioprobench/blob/main/Metrics/REA-ERR.py): legacy evaluator · `Metrics/REA-ERR.py`

### Portable evaluation skill

Preferred integration surface.

- [Skill contract](https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/SKILL.md): agent package · Portable evaluator scope and agent registration. · `skills/evaluate-protocol-outputs/SKILL.md`
- [Normalize payloads](https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/scripts/normalize.py): normalization layer · Canonicalizes heterogeneous prediction and reference payloads. · `skills/evaluate-protocol-outputs/scripts/normalize.py`
- [Evaluation dispatcher](https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/scripts/evaluate.py): skill entry point · Routes normalized inputs by task. · `skills/evaluate-protocol-outputs/scripts/evaluate.py`
- [Task evaluators](https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/scripts/generation.py): semantic scoring · Generation, QA, ordering, and error-detection semantics. · `skills/evaluate-protocol-outputs/scripts/generation.py`
- [Judge consistency](https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/scripts/judge_consistency.py): reasoning evaluator · Optional judge-based reasoning consistency assessment. · `skills/evaluate-protocol-outputs/scripts/judge_consistency.py`
- [Metric profile](https://github.com/yuyangsunshine/bioprobench/blob/main/skills/evaluate-protocol-outputs/references/metric-definitions.md): evaluation specification · Benchmark-compatible metric definitions and semantics. · `skills/evaluate-protocol-outputs/references/metric-definitions.md`

## Connections

- Task test datasets → Remote inference: task inputs
- Task test datasets → Local inference: task inputs
- Task contract → Prompt construction: formats prompts
- Prompt construction → Remote inference: prompts
- Prompt construction → Local inference: prompts
- Remote inference → LLM API endpoints: invokes
- Local inference → HF models and runtime: loads
- Remote inference → Response JSON: writes
- Local inference → Response JSON: writes
- Response JSON → QA metrics: offline scoring
- Response JSON → Ordering metrics: offline scoring
- Response JSON → Error metrics: offline scoring
- Response JSON → Generation metrics: offline scoring
- Response JSON → Reasoning consistency: offline scoring
- Response JSON → Normalize payloads: predictions and references
- Normalize payloads → Evaluation dispatcher: normalized records
- Evaluation dispatcher → Task evaluators: dispatches by task
- Evaluation dispatcher → Judge consistency: when applicable
- Metric profile → Task evaluators: defines semantics
- Skill contract → Evaluation dispatcher: packages
- Task contract → Normalize payloads: input conventions