How google/big-bench works
BIG-bench is a collaborative benchmark for evaluating language models on a broad collection of JSON-defined and programmatic tasks. The main workflow is to select a task and model, run evaluation, and record scores and query data; a separate SeqIO path registers JSON tasks and mixtures for dataset use. Results can also feed benchmark summaries and leaderboard artifacts. The graph combines excerpted calls with documented capabilities; dashed edges mark relationships whose implementation was not shown in the excerpts.
Main entry points: model.py Model interface · evaluate_task.py Evaluation runner · bigbench_bridge.py BIG-bench bridge · query_logging_model.py Query logger
Components 23
Benchmark tasks
- Task collection
bigbench/benchmark_tasks - JSON task runtime
bigbench/api/json_task.py - Programmatic tasks
bigbench/api/task.py
Evaluation API
- Evaluation runner
bigbench/evaluate_task.py - Task utilities
bigbench/api/util.py - Task metrics
bigbench/api/task_metrics.py
Model implementations
- Model interface
bigbench/api/model.py - Hugging Face models
bigbench/models/huggingface_models.py - Human model
bigbench/models/human_model.py - Dummy model
bigbench/models/dummy_model.py - JSON-RPC model
bigbench/models/json_rpc_model.py - Turing generator
bigbench/models/turing_text_generator.py - Model utilities
bigbench/models/model_utils.py - Query logger
bigbench/models/query_logging_model.py
SeqIO integration
- SeqIO registration
bigbench/bbseqio/tasks.py - SeqIO task API
bigbench/bbseqio/task_api.py - BIG-bench bridge
bigbench/bbseqio/bigbench_bridge.py - JSON task paths
bigbench/bbseqio/bigbench_json_paths.py - BIG-bench Lite
bigbench/bbseqio/bigbench_lite.py
Results and reporting
- Evaluation results
bigbench/api/results.py - Task summaries
bigbench/task_postprocessing_scripts/generate_task_summaries.py
Other components
- Benchmark user
- SeqIO
Connections 24
- Benchmark user → Evaluation runner: runs evaluation
README.md - Task collection → JSON task runtime: defines JSON tasks
README.md - Task collection → Programmatic tasks: defines programmatic tasks
README.md - Evaluation runner → Query logger: wraps model
bigbench/evaluate_task.py - Evaluation runner → Evaluation results: writes results
bigbench/evaluate_task.py - Evaluation runner → Model interface: uses model interface
bigbench/evaluate_task.py - JSON task runtime → Task metrics: uses metrics
bigbench/api/json_task.py - Programmatic tasks → Model interface: defines model queries
bigbench/api/task.py - Query logger → Model interface: delegates queries
bigbench/models/query_logging_model.py - Query logger → Model utilities: postprocesses output
bigbench/models/query_logging_model.py - Hugging Face models → Model interface: implements interface
bigbench/models/huggingface_models.py - Human model → Model interface: implements interface
bigbench/models/human_model.py - Dummy model → Model interface: implements interface
bigbench/models/dummy_model.py - JSON-RPC model → Model interface: implements interface
bigbench/models/json_rpc_model.py - Dummy model → Turing generator: uses generator
bigbench/models/dummy_model.py - Hugging Face models → Model utilities: uses utilities
bigbench/models/huggingface_models.py - Evaluation results → Task summaries: summarizes scores
bigbench/task_postprocessing_scripts/generate_task_summaries.py - SeqIO registration → SeqIO task API: registers tasks
bigbench/bbseqio/tasks.py - SeqIO task API → BIG-bench bridge: configures bridge
bigbench/bbseqio/task_api.py - SeqIO task API → SeqIO: registers tasks
bigbench/bbseqio/task_api.py - BIG-bench bridge → JSON task runtime: loads JSON tasks
bigbench/bbseqio/bigbench_bridge.py - BIG-bench bridge → Model utilities: postprocesses output
bigbench/bbseqio/bigbench_bridge.py - SeqIO registration → BIG-bench Lite: registers Lite mixture
bigbench/bbseqio/tasks.py - JSON task paths → BIG-bench bridge: resolves task paths
bigbench/bbseqio/bigbench_json_paths.py