# allenai/ifbench architecture

> Architecture diagram of the GitHub repository allenai/ifbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/allenai/ifbench
- Repository: https://github.com/allenai/ifbench

## Overview

IFBench is a benchmark and Python verifier library for measuring precise instruction following. Its main workflow loads benchmark prompts, optionally generates model responses through an OpenAI-compatible API, then evaluates each response against registered constraints and writes strict and loose accuracy results. The package provides both classic IFEval and IFBench OOD verifiers, backed by shared text utilities. The README also describes multiturn and IF-RLVR training data/capabilities; their implementations are not present in the sampled source, so those links are documented but unverified here.

Main entry points: [`generate_responses.py`](https://github.com/allenai/ifbench/blob/main/generate_responses.py) (Response generator), [`ifbench/__init__.py`](https://github.com/allenai/ifbench/blob/main/ifbench/__init__.py) (Python package API), [`evaluation_lib.py`](https://github.com/allenai/ifbench/blob/main/evaluation_lib.py) (Scoring and reports), [`ifbench/instructions_registry.py`](https://github.com/allenai/ifbench/blob/main/ifbench/instructions_registry.py) (Verifier registry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Benchmark workflow"]
  node_generator["Response generator"]
  node_settings["Run settings<br/>[config.py]"]
  node_response_file["Prompt responses"]
  node_eval_cli["Evaluation command<br/>[run_eval.py]"]
  node_evaluation["Scoring and reports<br/>[evaluation_lib.py]"]
end

subgraph group_verification["Verification library"]
  node_registry["Verifier registry"]
  node_ood_verifiers["IFBench OOD verifiers<br/>[instructions.py]"]
  node_classic_verifiers["Classic IFEval verifiers"]
  node_text_util["Text utilities"]
  node_package_api["Python package API<br/>[__init__.py]"]
end

subgraph group_assets["Benchmark assets"]
  node_dataset["IFBench test data<br/>[IFBench_test.jsonl]"]
  node_multiturn_data["Multiturn test data"]
end

subgraph group_training["Training ecosystem"]
  node_training_data["IF-RLVR training data"]
  node_open_instruct["Open-Instruct training"]
end

node_researcher(("Benchmark user"))
node_model_api["OpenAI-compatible API"]

node_researcher -->|"runs generation"| node_generator
node_generator -->|"loads settings"| node_settings
node_generator -->|"loads prompts"| node_dataset
node_generator -->|"requests completions"| node_model_api
node_generator -->|"writes responses"| node_response_file
node_researcher -->|"runs evaluation"| node_eval_cli
node_eval_cli -->|"calls scoring"| node_evaluation
node_evaluation -->|"reads prompts"| node_dataset
node_response_file -->|"supplies completions"| node_evaluation
node_evaluation -->|"looks up verifiers"| node_registry
node_registry -->|"registers classes"| node_ood_verifiers
node_registry -->|"registers classes"| node_classic_verifiers
node_ood_verifiers -->|"uses helpers"| node_text_util
node_classic_verifiers -->|"uses helpers"| node_text_util
node_package_api -->|"exposes"| node_ood_verifiers
node_package_api -->|"exposes"| node_classic_verifiers
node_package_api -->|"exposes"| node_registry
node_package_api -->|"exposes"| node_text_util
node_researcher -.->|"can evaluate"| node_multiturn_data
node_training_data -.->|"trains with"| node_open_instruct

click node_generator "https://github.com/allenai/ifbench/blob/main/generate_responses.py"
click node_settings "https://github.com/allenai/ifbench/blob/main/config.py"
click node_eval_cli "https://github.com/allenai/ifbench/blob/main/run_eval.py"
click node_evaluation "https://github.com/allenai/ifbench/blob/main/evaluation_lib.py"
click node_dataset "https://github.com/allenai/ifbench/blob/main/ifbench/data/IFBench_test.jsonl"
click node_registry "https://github.com/allenai/ifbench/blob/main/ifbench/instructions_registry.py"
click node_ood_verifiers "https://github.com/allenai/ifbench/blob/main/ifbench/instructions.py"
click node_classic_verifiers "https://github.com/allenai/ifbench/blob/main/ifbench/classic_instructions.py"
click node_text_util "https://github.com/allenai/ifbench/blob/main/ifbench/instructions_util.py"
click node_package_api "https://github.com/allenai/ifbench/blob/main/ifbench/__init__.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generator,node_settings,node_response_file,node_eval_cli,node_evaluation,node_researcher toneBlue
class node_registry,node_ood_verifiers,node_classic_verifiers,node_text_util,node_package_api toneAmber
class node_dataset,node_multiturn_data,node_model_api toneMint
class node_training_data,node_open_instruct toneRose
```

## Components

### Benchmark workflow

- [Response generator](https://github.com/allenai/ifbench/blob/main/generate_responses.py): `generate_responses.py`
- [Run settings](https://github.com/allenai/ifbench/blob/main/config.py): `config.py`
- **Prompt responses**
- [Evaluation command](https://github.com/allenai/ifbench/blob/main/run_eval.py): `run_eval.py`
- [Scoring and reports](https://github.com/allenai/ifbench/blob/main/evaluation_lib.py): `evaluation_lib.py`

### Verification library

- [Verifier registry](https://github.com/allenai/ifbench/blob/main/ifbench/instructions_registry.py): `ifbench/instructions_registry.py`
- [IFBench OOD verifiers](https://github.com/allenai/ifbench/blob/main/ifbench/instructions.py): `ifbench/instructions.py`
- [Classic IFEval verifiers](https://github.com/allenai/ifbench/blob/main/ifbench/classic_instructions.py): `ifbench/classic_instructions.py`
- [Text utilities](https://github.com/allenai/ifbench/blob/main/ifbench/instructions_util.py): `ifbench/instructions_util.py`
- [Python package API](https://github.com/allenai/ifbench/blob/main/ifbench/__init__.py): `ifbench/__init__.py`

### Benchmark assets

- [IFBench test data](https://github.com/allenai/ifbench/blob/main/ifbench/data/IFBench_test.jsonl): `ifbench/data/IFBench_test.jsonl`
- **Multiturn test data**

### Training ecosystem

- **IF-RLVR training data**
- **Open-Instruct training**

### Other components

- **Benchmark user**
- **OpenAI-compatible API**

## Connections

- Benchmark user → Response generator: runs generation (evidence: `generate_responses.py`)
- Response generator → Run settings: loads settings (evidence: `generate_responses.py`)
- Response generator → IFBench test data: loads prompts (evidence: `generate_responses.py`)
- Response generator → OpenAI-compatible API: requests completions (evidence: `generate_responses.py`)
- Response generator → Prompt responses: writes responses (evidence: `generate_responses.py`)
- Benchmark user → Evaluation command: runs evaluation (evidence: `README.md`)
- Evaluation command → Scoring and reports: calls scoring (evidence: `run_eval.py`)
- Scoring and reports → IFBench test data: reads prompts (evidence: `README.md`)
- Prompt responses → Scoring and reports: supplies completions (evidence: `README.md`)
- Scoring and reports → Verifier registry: looks up verifiers (evidence: `evaluation_lib.py`)
- Verifier registry → IFBench OOD verifiers: registers classes (evidence: `ifbench/instructions_registry.py`)
- Verifier registry → Classic IFEval verifiers: registers classes (evidence: `ifbench/instructions_registry.py`)
- IFBench OOD verifiers → Text utilities: uses helpers (evidence: `ifbench/instructions.py`)
- Classic IFEval verifiers → Text utilities: uses helpers (evidence: `ifbench/classic_instructions.py`)
- Python package API → IFBench OOD verifiers: exposes (evidence: `ifbench/__init__.py`)
- Python package API → Classic IFEval verifiers: exposes (evidence: `ifbench/__init__.py`)
- Python package API → Verifier registry: exposes (evidence: `ifbench/__init__.py`)
- Python package API → Text utilities: exposes (evidence: `ifbench/__init__.py`)
- Benchmark user → Multiturn test data: can evaluate (evidence: `README.md`)
- IF-RLVR training data → Open-Instruct training: trains with (evidence: `README.md`)