allenai/ifbench

How allenai/ifbench works

IFBench is a benchmark and Python verifier library for measuring precise instruction following. Its main workflow loads benchmark prompts, optionally generates model responses through an OpenAI-compatible API, then evaluates each response against registered constraints and writes strict and loose accuracy results. The package provides both classic IFEval and IFBench OOD verifiers, backed by shared text utilities. The README also describes multiturn and IF-RLVR training data/capabilities; their implementations are not present in the sampled source, so those links are documented but unverified here.

Main entry points: generate_responses.py Response generator · __init__.py Python package API · evaluation_lib.py Scoring and reports · instructions_registry.py Verifier registry

Components 16

Benchmark workflow

Response generator generate_responses.py
Run settings config.py
Prompt responses
Evaluation command run_eval.py
Scoring and reports evaluation_lib.py

Verification library

Verifier registry ifbench/instructions_registry.py
IFBench OOD verifiers ifbench/instructions.py
Classic IFEval verifiers ifbench/classic_instructions.py
Text utilities ifbench/instructions_util.py
Python package API ifbench/__init__.py

Benchmark assets

IFBench test data ifbench/data/IFBench_test.jsonl
Multiturn test data

Training ecosystem

IF-RLVR training data
Open-Instruct training

Other components

Benchmark user
OpenAI-compatible API

Connections 20

Benchmark user → Response generator: runs generation generate_responses.py
Response generator → Run settings: loads settings generate_responses.py
Response generator → IFBench test data: loads prompts generate_responses.py
Response generator → OpenAI-compatible API: requests completions generate_responses.py
Response generator → Prompt responses: writes responses generate_responses.py
Benchmark user → Evaluation command: runs evaluation README.md
Evaluation command → Scoring and reports: calls scoring run_eval.py
Scoring and reports → IFBench test data: reads prompts README.md
Prompt responses → Scoring and reports: supplies completions README.md
Scoring and reports → Verifier registry: looks up verifiers evaluation_lib.py
Verifier registry → IFBench OOD verifiers: registers classes ifbench/instructions_registry.py
Verifier registry → Classic IFEval verifiers: registers classes ifbench/instructions_registry.py
IFBench OOD verifiers → Text utilities: uses helpers ifbench/instructions.py
Classic IFEval verifiers → Text utilities: uses helpers ifbench/classic_instructions.py
Python package API → IFBench OOD verifiers: exposes ifbench/__init__.py
Python package API → Classic IFEval verifiers: exposes ifbench/__init__.py
Python package API → Verifier registry: exposes ifbench/__init__.py
Python package API → Text utilities: exposes ifbench/__init__.py
Benchmark user → Multiturn test data: can evaluate README.md
IF-RLVR training data → Open-Instruct training: trains with README.md

Drawn by GitDiagram on November 13, 2026 from the repository's file tree, README and source files. Read as Markdown ·