# kaggle/kaggle-benchmarks architecture

> Architecture diagram of the GitHub repository kaggle/kaggle-benchmarks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kaggle/kaggle-benchmarks
- Repository: https://github.com/kaggle/kaggle-benchmarks

## Overview

Kaggle Benchmarks is a Python framework for defining and running evaluations of language models. The principal workflow is: a user defines a task, runs it over parameters or dataset rows, the task interacts with a model, assertions validate results, and runs can be recorded for review. The library also supports structured and multimodal inputs, tool use, parallel evaluation, and interactive notebook views. Sampled source confirms task registration, chat creation, schema processing, event dispatch, and Kaggle file persistence; some integration wiring is not shown, so uncertain links are omitted.

Main entry points: [`src/kaggle_benchmarks/tasks.py`](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/tasks.py) (Task definition and run), [`src/kaggle_benchmarks/actors/llms.py`](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/actors/llms.py) (LLM prompt interface), [`src/kaggle_benchmarks/kaggle/client.py`](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/kaggle/client.py) (Kaggle task and run files), [`src/kaggle_benchmarks/chats.py`](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/chats.py) (Chat lifecycle)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_evaluation["Evaluation"]
  node_task["Task definition and run<br/>[tasks.py]"]
  node_scheduling["Dataset and grid evaluation<br/>[scheduling.py]"]
  node_assertions["Output assertions<br/>[assertions.py]"]
  node_results["Result types<br/>[results.py]"]
  node_runs["Run records<br/>[runs.py]"]
end

subgraph group_model_io["Model Interaction"]
  node_llm["LLM prompt interface<br/>[llms.py]"]
  node_prompting["Schema prompting<br/>[prompting.py]"]
  node_content["Multimodal content"]
  node_tools["Model tools"]
  node_environment["Execution environments"]
end

subgraph group_conversation["Conversation State"]
  node_chats["Chat lifecycle<br/>[chats.py]"]
  node_messages["Conversation messages<br/>[messages.py]"]
  node_llm_messages["LLM responses and streams<br/>[llm_messages.py]"]
  node_events["Runtime events<br/>[events.py]"]
end

subgraph group_integration["Persistence and UI"]
  node_clients["Client selection<br/>[clients.py]"]
  node_kaggle_client["Kaggle task and run files<br/>[client.py]"]
  node_run_files["Benchmark file outputs"]
  node_panel["Notebook visualization<br/>[panel.py]"]
end

node_user(("Benchmark author"))
node_wikipedia(("Wikipedia search service"))

node_user -->|"defines and runs"| node_task
node_task -->|"opens chat"| node_chats
node_task -->|"creates run"| node_runs
node_task -->|"registers task"| node_clients
node_task -->|"checks output"| node_assertions
node_scheduling -->|"evaluates function"| node_task
node_scheduling -->|"returns results"| node_user
node_llm -->|"processes schema"| node_prompting
node_llm -->|"accepts media"| node_content
node_llm -->|"builds history"| node_messages
node_llm -->|"returns response"| node_llm_messages
node_llm -->|"accepts tools"| node_tools
node_tools -.->|"executes code"| node_environment
node_chats -->|"tracks messages"| node_messages
node_llm_messages -->|"dispatches stream events"| node_events
node_clients -->|"selects client"| node_kaggle_client
node_kaggle_client -->|"writes and reads"| node_run_files
node_kaggle_client -->|"stores run data"| node_runs
node_panel -->|"renders chats"| node_chats
node_panel -->|"renders results"| node_runs
node_wikipedia -.->|"provides search data"| node_user

click node_task "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/tasks.py"
click node_scheduling "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/orchestration/scheduling.py"
click node_assertions "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/assertions.py"
click node_results "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/results.py"
click node_runs "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/runs.py"
click node_llm "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/actors/llms.py"
click node_prompting "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/prompting.py"
click node_content "https://github.com/kaggle/kaggle-benchmarks/tree/ci/src/kaggle_benchmarks/content_types"
click node_tools "https://github.com/kaggle/kaggle-benchmarks/tree/ci/src/kaggle_benchmarks/tools"
click node_environment "https://github.com/kaggle/kaggle-benchmarks/tree/ci/src/kaggle_benchmarks/envs"
click node_chats "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/chats.py"
click node_messages "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/messages.py"
click node_llm_messages "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/llm_messages.py"
click node_events "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/events.py"
click node_clients "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/clients.py"
click node_kaggle_client "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/kaggle/client.py"
click node_panel "https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/ui/panel.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_task,node_scheduling,node_assertions,node_results,node_runs toneBlue
class node_llm,node_prompting,node_content,node_tools,node_environment toneAmber
class node_chats,node_messages,node_llm_messages,node_events toneMint
class node_clients,node_kaggle_client,node_run_files,node_panel toneRose
class node_user,node_wikipedia toneIndigo
```

## Components

### Evaluation

- [Task definition and run](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/tasks.py): `src/kaggle_benchmarks/tasks.py`
- [Dataset and grid evaluation](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/orchestration/scheduling.py): `src/kaggle_benchmarks/orchestration/scheduling.py`
- [Output assertions](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/assertions.py): `src/kaggle_benchmarks/assertions.py`
- [Result types](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/results.py): `src/kaggle_benchmarks/results.py`
- [Run records](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/runs.py): `src/kaggle_benchmarks/runs.py`

### Model Interaction

- [LLM prompt interface](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/actors/llms.py): `src/kaggle_benchmarks/actors/llms.py`
- [Schema prompting](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/prompting.py): `src/kaggle_benchmarks/prompting.py`
- [Multimodal content](https://github.com/kaggle/kaggle-benchmarks/tree/ci/src/kaggle_benchmarks/content_types): `src/kaggle_benchmarks/content_types`
- [Model tools](https://github.com/kaggle/kaggle-benchmarks/tree/ci/src/kaggle_benchmarks/tools): `src/kaggle_benchmarks/tools`
- [Execution environments](https://github.com/kaggle/kaggle-benchmarks/tree/ci/src/kaggle_benchmarks/envs): `src/kaggle_benchmarks/envs`

### Conversation State

- [Chat lifecycle](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/chats.py): `src/kaggle_benchmarks/chats.py`
- [Conversation messages](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/messages.py): `src/kaggle_benchmarks/messages.py`
- [LLM responses and streams](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/llm_messages.py): `src/kaggle_benchmarks/llm_messages.py`
- [Runtime events](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/events.py): `src/kaggle_benchmarks/events.py`

### Persistence and UI

- [Client selection](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/clients.py): `src/kaggle_benchmarks/clients.py`
- [Kaggle task and run files](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/kaggle/client.py): `src/kaggle_benchmarks/kaggle/client.py`
- **Benchmark file outputs**
- [Notebook visualization](https://github.com/kaggle/kaggle-benchmarks/blob/ci/src/kaggle_benchmarks/ui/panel.py): `src/kaggle_benchmarks/ui/panel.py`

### Other components

- **Benchmark author**
- **Wikipedia search service**

## Connections

- Benchmark author → Task definition and run: defines and runs
- Task definition and run → Chat lifecycle: opens chat
- Task definition and run → Run records: creates run
- Task definition and run → Client selection: registers task
- Task definition and run → Output assertions: checks output
- Dataset and grid evaluation → Task definition and run: evaluates function
- Dataset and grid evaluation → Benchmark author: returns results
- LLM prompt interface → Schema prompting: processes schema
- LLM prompt interface → Multimodal content: accepts media
- LLM prompt interface → Conversation messages: builds history
- LLM prompt interface → LLM responses and streams: returns response
- LLM prompt interface → Model tools: accepts tools
- Model tools → Execution environments: executes code
- Chat lifecycle → Conversation messages: tracks messages
- LLM responses and streams → Runtime events: dispatches stream events
- Client selection → Kaggle task and run files: selects client
- Kaggle task and run files → Benchmark file outputs: writes and reads
- Kaggle task and run files → Run records: stores run data
- Notebook visualization → Chat lifecycle: renders chats
- Notebook visualization → Run records: renders results
- Wikipedia search service → Benchmark author: provides search data