# google/big-bench architecture

> Architecture diagram of the GitHub repository google/big-bench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google/big-bench
- Repository: https://github.com/google/big-bench

## Overview

BIG-bench is a collaborative benchmark for evaluating language models on a broad collection of JSON-defined and programmatic tasks. The main workflow is to select a task and model, run evaluation, and record scores and query data; a separate SeqIO path registers JSON tasks and mixtures for dataset use. Results can also feed benchmark summaries and leaderboard artifacts. The graph combines excerpted calls with documented capabilities; dashed edges mark relationships whose implementation was not shown in the excerpts.

Main entry points: [`bigbench/api/model.py`](https://github.com/google/big-bench/blob/main/bigbench/api/model.py) (Model interface), [`bigbench/evaluate_task.py`](https://github.com/google/big-bench/blob/main/bigbench/evaluate_task.py) (Evaluation runner), [`bigbench/bbseqio/bigbench_bridge.py`](https://github.com/google/big-bench/blob/main/bigbench/bbseqio/bigbench_bridge.py) (BIG-bench bridge), [`bigbench/models/query_logging_model.py`](https://github.com/google/big-bench/blob/main/bigbench/models/query_logging_model.py) (Query logger)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_tasks["Benchmark tasks"]
  node_task_collection["Task collection"]
  node_json_task["JSON task runtime<br/>[json_task.py]"]
  node_programmatic_task["Programmatic tasks<br/>[task.py]"]
end

subgraph group_evaluation["Evaluation API"]
  node_evaluate["Evaluation runner<br/>[evaluate_task.py]"]
  node_api_utils["Task utilities<br/>[util.py]"]
  node_metrics["Task metrics<br/>[task_metrics.py]"]
end

subgraph group_models["Model implementations"]
  node_model_api["Model interface<br/>[model.py]"]
  node_hf_model["Hugging Face models"]
  node_human_model["Human model<br/>[human_model.py]"]
  node_dummy_model["Dummy model<br/>[dummy_model.py]"]
  node_rpc_model["JSON-RPC model<br/>[json_rpc_model.py]"]
  node_turing_generator["Turing generator"]
  node_model_utils["Model utilities<br/>[model_utils.py]"]
  node_query_logger["Query logger"]
end

subgraph group_seqio["SeqIO integration"]
  node_seqio_tasks["SeqIO registration<br/>[tasks.py]"]
  node_seqio_api["SeqIO task API<br/>[task_api.py]"]
  node_seqio_bridge["BIG-bench bridge<br/>[bigbench_bridge.py]"]
  node_json_paths["JSON task paths"]
  node_lite["BIG-bench Lite<br/>[bigbench_lite.py]"]
end

subgraph group_reporting["Results and reporting"]
  node_results["Evaluation results<br/>[results.py]"]
  node_summaries["Task summaries"]
end

node_user(("Benchmark user"))
node_seqio{{"SeqIO"}}

node_user -.->|"runs evaluation"| node_evaluate
node_task_collection -.->|"defines JSON tasks"| node_json_task
node_task_collection -.->|"defines programmatic tasks"| node_programmatic_task
node_evaluate -->|"wraps model"| node_query_logger
node_evaluate -->|"writes results"| node_results
node_evaluate -.->|"uses model interface"| node_model_api
node_json_task -.->|"uses metrics"| node_metrics
node_programmatic_task -.->|"defines model queries"| node_model_api
node_query_logger -->|"delegates queries"| node_model_api
node_query_logger -->|"postprocesses output"| node_model_utils
node_hf_model -.->|"implements interface"| node_model_api
node_human_model -.->|"implements interface"| node_model_api
node_dummy_model -.->|"implements interface"| node_model_api
node_rpc_model -.->|"implements interface"| node_model_api
node_dummy_model -.->|"uses generator"| node_turing_generator
node_hf_model -.->|"uses utilities"| node_model_utils
node_results -.->|"summarizes scores"| node_summaries
node_seqio_tasks -->|"registers tasks"| node_seqio_api
node_seqio_api -->|"configures bridge"| node_seqio_bridge
node_seqio_api -->|"registers tasks"| node_seqio
node_seqio_bridge -->|"loads JSON tasks"| node_json_task
node_seqio_bridge -->|"postprocesses output"| node_model_utils
node_seqio_tasks -.->|"registers Lite mixture"| node_lite
node_json_paths -.->|"resolves task paths"| node_seqio_bridge

click node_task_collection "https://github.com/google/big-bench/tree/main/bigbench/benchmark_tasks"
click node_json_task "https://github.com/google/big-bench/blob/main/bigbench/api/json_task.py"
click node_programmatic_task "https://github.com/google/big-bench/blob/main/bigbench/api/task.py"
click node_evaluate "https://github.com/google/big-bench/blob/main/bigbench/evaluate_task.py"
click node_api_utils "https://github.com/google/big-bench/blob/main/bigbench/api/util.py"
click node_metrics "https://github.com/google/big-bench/blob/main/bigbench/api/task_metrics.py"
click node_model_api "https://github.com/google/big-bench/blob/main/bigbench/api/model.py"
click node_hf_model "https://github.com/google/big-bench/blob/main/bigbench/models/huggingface_models.py"
click node_human_model "https://github.com/google/big-bench/blob/main/bigbench/models/human_model.py"
click node_dummy_model "https://github.com/google/big-bench/blob/main/bigbench/models/dummy_model.py"
click node_rpc_model "https://github.com/google/big-bench/blob/main/bigbench/models/json_rpc_model.py"
click node_turing_generator "https://github.com/google/big-bench/blob/main/bigbench/models/turing_text_generator.py"
click node_model_utils "https://github.com/google/big-bench/blob/main/bigbench/models/model_utils.py"
click node_query_logger "https://github.com/google/big-bench/blob/main/bigbench/models/query_logging_model.py"
click node_results "https://github.com/google/big-bench/blob/main/bigbench/api/results.py"
click node_summaries "https://github.com/google/big-bench/blob/main/bigbench/task_postprocessing_scripts/generate_task_summaries.py"
click node_seqio_tasks "https://github.com/google/big-bench/blob/main/bigbench/bbseqio/tasks.py"
click node_seqio_api "https://github.com/google/big-bench/blob/main/bigbench/bbseqio/task_api.py"
click node_seqio_bridge "https://github.com/google/big-bench/blob/main/bigbench/bbseqio/bigbench_bridge.py"
click node_json_paths "https://github.com/google/big-bench/blob/main/bigbench/bbseqio/bigbench_json_paths.py"
click node_lite "https://github.com/google/big-bench/blob/main/bigbench/bbseqio/bigbench_lite.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_task_collection,node_json_task,node_programmatic_task,node_user toneBlue
class node_evaluate,node_api_utils,node_metrics toneAmber
class node_model_api,node_hf_model,node_human_model,node_dummy_model,node_rpc_model,node_turing_generator,node_model_utils,node_query_logger toneMint
class node_seqio_tasks,node_seqio_api,node_seqio_bridge,node_json_paths,node_lite toneRose
class node_results,node_summaries,node_seqio toneIndigo
```

## Components

### Benchmark tasks

- [Task collection](https://github.com/google/big-bench/tree/main/bigbench/benchmark_tasks): `bigbench/benchmark_tasks`
- [JSON task runtime](https://github.com/google/big-bench/blob/main/bigbench/api/json_task.py): `bigbench/api/json_task.py`
- [Programmatic tasks](https://github.com/google/big-bench/blob/main/bigbench/api/task.py): `bigbench/api/task.py`

### Evaluation API

- [Evaluation runner](https://github.com/google/big-bench/blob/main/bigbench/evaluate_task.py): `bigbench/evaluate_task.py`
- [Task utilities](https://github.com/google/big-bench/blob/main/bigbench/api/util.py): `bigbench/api/util.py`
- [Task metrics](https://github.com/google/big-bench/blob/main/bigbench/api/task_metrics.py): `bigbench/api/task_metrics.py`

### Model implementations

- [Model interface](https://github.com/google/big-bench/blob/main/bigbench/api/model.py): `bigbench/api/model.py`
- [Hugging Face models](https://github.com/google/big-bench/blob/main/bigbench/models/huggingface_models.py): `bigbench/models/huggingface_models.py`
- [Human model](https://github.com/google/big-bench/blob/main/bigbench/models/human_model.py): `bigbench/models/human_model.py`
- [Dummy model](https://github.com/google/big-bench/blob/main/bigbench/models/dummy_model.py): `bigbench/models/dummy_model.py`
- [JSON-RPC model](https://github.com/google/big-bench/blob/main/bigbench/models/json_rpc_model.py): `bigbench/models/json_rpc_model.py`
- [Turing generator](https://github.com/google/big-bench/blob/main/bigbench/models/turing_text_generator.py): `bigbench/models/turing_text_generator.py`
- [Model utilities](https://github.com/google/big-bench/blob/main/bigbench/models/model_utils.py): `bigbench/models/model_utils.py`
- [Query logger](https://github.com/google/big-bench/blob/main/bigbench/models/query_logging_model.py): `bigbench/models/query_logging_model.py`

### SeqIO integration

- [SeqIO registration](https://github.com/google/big-bench/blob/main/bigbench/bbseqio/tasks.py): `bigbench/bbseqio/tasks.py`
- [SeqIO task API](https://github.com/google/big-bench/blob/main/bigbench/bbseqio/task_api.py): `bigbench/bbseqio/task_api.py`
- [BIG-bench bridge](https://github.com/google/big-bench/blob/main/bigbench/bbseqio/bigbench_bridge.py): `bigbench/bbseqio/bigbench_bridge.py`
- [JSON task paths](https://github.com/google/big-bench/blob/main/bigbench/bbseqio/bigbench_json_paths.py): `bigbench/bbseqio/bigbench_json_paths.py`
- [BIG-bench Lite](https://github.com/google/big-bench/blob/main/bigbench/bbseqio/bigbench_lite.py): `bigbench/bbseqio/bigbench_lite.py`

### Results and reporting

- [Evaluation results](https://github.com/google/big-bench/blob/main/bigbench/api/results.py): `bigbench/api/results.py`
- [Task summaries](https://github.com/google/big-bench/blob/main/bigbench/task_postprocessing_scripts/generate_task_summaries.py): `bigbench/task_postprocessing_scripts/generate_task_summaries.py`

### Other components

- **Benchmark user**
- **SeqIO**

## Connections

- Benchmark user → Evaluation runner: runs evaluation (evidence: `README.md`)
- Task collection → JSON task runtime: defines JSON tasks (evidence: `README.md`)
- Task collection → Programmatic tasks: defines programmatic tasks (evidence: `README.md`)
- Evaluation runner → Query logger: wraps model (evidence: `bigbench/evaluate_task.py`)
- Evaluation runner → Evaluation results: writes results (evidence: `bigbench/evaluate_task.py`)
- Evaluation runner → Model interface: uses model interface (evidence: `bigbench/evaluate_task.py`)
- JSON task runtime → Task metrics: uses metrics (evidence: `bigbench/api/json_task.py`)
- Programmatic tasks → Model interface: defines model queries (evidence: `bigbench/api/task.py`)
- Query logger → Model interface: delegates queries (evidence: `bigbench/models/query_logging_model.py`)
- Query logger → Model utilities: postprocesses output (evidence: `bigbench/models/query_logging_model.py`)
- Hugging Face models → Model interface: implements interface (evidence: `bigbench/models/huggingface_models.py`)
- Human model → Model interface: implements interface (evidence: `bigbench/models/human_model.py`)
- Dummy model → Model interface: implements interface (evidence: `bigbench/models/dummy_model.py`)
- JSON-RPC model → Model interface: implements interface (evidence: `bigbench/models/json_rpc_model.py`)
- Dummy model → Turing generator: uses generator (evidence: `bigbench/models/dummy_model.py`)
- Hugging Face models → Model utilities: uses utilities (evidence: `bigbench/models/huggingface_models.py`)
- Evaluation results → Task summaries: summarizes scores (evidence: `bigbench/task_postprocessing_scripts/generate_task_summaries.py`)
- SeqIO registration → SeqIO task API: registers tasks (evidence: `bigbench/bbseqio/tasks.py`)
- SeqIO task API → BIG-bench bridge: configures bridge (evidence: `bigbench/bbseqio/task_api.py`)
- SeqIO task API → SeqIO: registers tasks (evidence: `bigbench/bbseqio/task_api.py`)
- BIG-bench bridge → JSON task runtime: loads JSON tasks (evidence: `bigbench/bbseqio/bigbench_bridge.py`)
- BIG-bench bridge → Model utilities: postprocesses output (evidence: `bigbench/bbseqio/bigbench_bridge.py`)
- SeqIO registration → BIG-bench Lite: registers Lite mixture (evidence: `bigbench/bbseqio/tasks.py`)
- JSON task paths → BIG-bench bridge: resolves task paths (evidence: `bigbench/bbseqio/bigbench_json_paths.py`)
