How moonshotai/perceptionbench works
Purpose and Entry Point
PerceptionBench evaluates atomic visual-perception ability in multimodal LLMs using a 3,000-question benchmark spanning ten capability categories. The executable entry point is eval/eval.py, run as python eval/eval.py; this is an evaluation harness rather than a service or web application.
Evaluation Inputs and Configuration
The evaluator reads all runtime settings from process environment or a dotenv file. .env.example defines the expected OpenAI-compatible API configuration, model identifiers, dataset location, generation controls, retry policy, and concurrency. The benchmark dataset defaults to a local JSONL path but is distributed externally through Hugging Face; it is not checked into this repository.
Relevant paths:
• .env.example
• eval/eval.py
• requirements.txt
Dataset-to-Prediction Stage
For each JSONL benchmark item, the evaluator supplies its question and associated image(s) to the configured target MLLM. The target endpoint must expose an OpenAI-compatible API and accept multimodal requests. Generation can be controlled through token limit, temperature, top-p/top-k, concurrent execution, and retries.
Primary boundary: benchmark content and image references enter the evaluator; model responses return from the external inference endpoint.
Relevant path:
• eval/eval.py
External Model Inference Runtime
Python is the runtime, with dependencies installed from requirements.txt. The evaluator connects to an externally hosted OpenAI-compatible model endpoint using OPENAI_API_KEY and OPENAI_BASE_URL. The evaluated MODEL is independently configurable from the judge model, allowing the same harness to score proprietary or self-hosted compatible models.
Relevant paths:
• eval/eval.py
• requirements.txt
• .env.example
LLM-Based Grading Stage
Predictions are not matched by simple string equality. Each candidate answer, reference answer, and grading instructions are sent to a separate LLM judge, defaulting to gpt-oss-120b. The judge prompt requires a strict per-item binary 0/1 verdict, handling valid free-form answer variants while maintaining benchmark scoring consistency.
Relevant paths:
• eval/judge_prompt.txt
• eval/eval.py
Aggregation and Result Artifacts
The evaluator persists per-item outputs to results/PerceptionBench_<model>.jsonl, then aggregates binary judgments into overall accuracy and per-category accuracy in results/PerceptionBench_<model>_scores.json. Result files are generated runtime artifacts, not present in the tracked tree.
Relevant path:
• eval/eval.py
Benchmark Scope and Supporting Assets
The benchmark isolates ten perception capabilities: visual relation, counting, attributes, depth/3D, localization, comparison, fine-grained recognition, contextual integration, OCR, and perception-related hallucination. Repository images support documentation; the benchmark paper records the benchmark methodology and reported frontier-model results.
Relevant paths:
• README.md
• paper/PerceptionBench.pdf
• images/fig_teaser.png
Components 12
Evaluation runtime
- Runtime environment · API endpoints, model IDs, dataset path, generation, retry, and concurrency settings.
.env.example - Python dependencies
requirements.txt - Evaluation harness · Runs dataset inference, LLM judging, persistence, and score aggregation.
eval/eval.py - Binary grading prompt · Instructs the judge to return a strict 0/1 verdict.
eval/judge_prompt.txt
External dependencies
- Benchmark JSONL and images · Externally distributed 3,000-question benchmark content and image references.
- Target MLLM endpoint · Configurable model under evaluation receives multimodal questions.
- LLM judge endpoint · Separately configurable judge, defaulting to gpt-oss-120b.
Generated results
- Per-item predictions and verdicts · Generated records for each benchmark item.
- Overall and category scores · Aggregated binary accuracy results.
Benchmark context
- Benchmark overview · Describes the ten perception capability categories.
README.md - Benchmark methodology
paper/PerceptionBench.pdf - Benchmark teaser image
Connections 12
- Runtime environment → Evaluation harness: configures
- Python dependencies → Evaluation harness: provides runtime
- Benchmark JSONL and images → Evaluation harness: JSONL items and images
- Evaluation harness → Target MLLM endpoint: multimodal inference requests
- Target MLLM endpoint → Evaluation harness: candidate responses
- Binary grading prompt → Evaluation harness: grading instructions
- Evaluation harness → LLM judge endpoint: candidate, reference, prompt
- LLM judge endpoint → Evaluation harness: binary verdicts
- Evaluation harness → Per-item predictions and verdicts: writes item outputs
- Per-item predictions and verdicts → Overall and category scores: aggregates verdicts
- Benchmark overview → Benchmark methodology: documents benchmark
- Benchmark teaser image → Benchmark overview: supports documentation