# confidence-routing/confidence-tool-routing architecture

> Architecture diagram of the GitHub repository confidence-routing/confidence-tool-routing, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/confidence-routing/confidence-tool-routing
- Repository: https://github.com/confidence-routing/confidence-tool-routing

## Overview

This repository is a measurement and evaluation harness for confidence-based tool routing, with dataset loaders, task grading, confidence estimators, routing analysis, and run reporting. The intended task-to-result flow is documented, but the README explicitly says the real model/tool runner is not yet implemented; the graph therefore does not invent that orchestration. Dataset-specific loaders and several evaluation relationships are included from the source index and excerpts, while unsampled implementation details are not inferred.

Main entry points: [`task_datasets/base.py`](https://github.com/confidence-routing/confidence-tool-routing/blob/main/task_datasets/base.py) (Dataset registry), [`eval_harness/models.py`](https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/models.py) (TaskRecord schema), [`eval_harness/metrics.py`](https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/metrics.py) (Evaluation metrics), [`eval_harness/router.py`](https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/router.py) (Routing and sweeps)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Task datasets"]
  node_dataset_base["Dataset registry<br/>[base.py]"]
  node_coqa["CoQA loader<br/>[coqa.py]"]
  node_gsm8k["GSM8K loader<br/>[gsm8k.py]"]
  node_headlines["Headlines loader<br/>[headlines.py]"]
  node_humaneval["HumanEval loader<br/>[humaneval.py]"]
  node_overruling["Overruling loader<br/>[overruling.py]"]
end

subgraph group_evaluation["Evaluation core"]
  node_task_record["TaskRecord schema<br/>[models.py]"]
  node_grading["Task grading<br/>[grading.py]"]
  node_metrics["Evaluation metrics<br/>[metrics.py]"]
  node_costs["Cost estimation<br/>[costs.py]"]
  node_report["Run reports<br/>[report.py]"]
end

subgraph group_decision["Confidence and routing"]
  node_confidence["Confidence estimators<br/>[confidence.py]"]
  node_router["Routing and sweeps<br/>[router.py]"]
end

subgraph group_runtime["Tools and persistence"]
  node_tools["Calculator and runner<br/>[tools.py]"]
  node_logger["JSONL run logger<br/>[logger.py]"]
end

node_researcher(("Experimenter"))
node_dataset_api["Dataset sources"]

node_researcher -->|"selects dataset"| node_dataset_base
node_dataset_base -->|"loads via registry"| node_coqa
node_dataset_base -->|"loads via registry"| node_gsm8k
node_dataset_base -->|"loads via registry"| node_headlines
node_dataset_base -->|"loads via registry"| node_humaneval
node_dataset_base -->|"loads via registry"| node_overruling
node_dataset_api -->|"provides rows"| node_headlines
node_dataset_api -->|"provides rows"| node_overruling
node_dataset_api -->|"provides benchmark"| node_humaneval
node_dataset_base -.->|"supplies task inputs"| node_task_record
node_task_record -->|"grades answers"| node_grading
node_grading -->|"runs code tests"| node_tools
node_task_record -->|"measures records"| node_metrics
node_metrics -->|"estimates costs"| node_costs
node_task_record -->|"serializes records"| node_logger
node_task_record -->|"stores confidence"| node_confidence
node_confidence -->|"supplies confidence"| node_router
node_task_record -->|"sets routing fields"| node_router
node_router -->|"grades sweeps"| node_metrics
node_task_record -->|"summarizes records"| node_report
node_metrics -->|"aggregates metrics"| node_report

click node_dataset_base "https://github.com/confidence-routing/confidence-tool-routing/blob/main/task_datasets/base.py"
click node_coqa "https://github.com/confidence-routing/confidence-tool-routing/blob/main/task_datasets/coqa.py"
click node_gsm8k "https://github.com/confidence-routing/confidence-tool-routing/blob/main/task_datasets/gsm8k.py"
click node_headlines "https://github.com/confidence-routing/confidence-tool-routing/blob/main/task_datasets/headlines.py"
click node_humaneval "https://github.com/confidence-routing/confidence-tool-routing/blob/main/task_datasets/humaneval.py"
click node_overruling "https://github.com/confidence-routing/confidence-tool-routing/blob/main/task_datasets/overruling.py"
click node_task_record "https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/models.py"
click node_grading "https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/grading.py"
click node_metrics "https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/metrics.py"
click node_costs "https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/costs.py"
click node_report "https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/report.py"
click node_confidence "https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/confidence.py"
click node_router "https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/router.py"
click node_tools "https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/tools.py"
click node_logger "https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/logger.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset_base,node_coqa,node_gsm8k,node_headlines,node_humaneval,node_overruling toneBlue
class node_task_record,node_grading,node_metrics,node_costs,node_report toneAmber
class node_confidence,node_router toneMint
class node_tools,node_logger toneRose
class node_researcher,node_dataset_api toneIndigo
```

## Components

### Task datasets

- [Dataset registry](https://github.com/confidence-routing/confidence-tool-routing/blob/main/task_datasets/base.py): `task_datasets/base.py`
- [CoQA loader](https://github.com/confidence-routing/confidence-tool-routing/blob/main/task_datasets/coqa.py): `task_datasets/coqa.py`
- [GSM8K loader](https://github.com/confidence-routing/confidence-tool-routing/blob/main/task_datasets/gsm8k.py): `task_datasets/gsm8k.py`
- [Headlines loader](https://github.com/confidence-routing/confidence-tool-routing/blob/main/task_datasets/headlines.py): `task_datasets/headlines.py`
- [HumanEval loader](https://github.com/confidence-routing/confidence-tool-routing/blob/main/task_datasets/humaneval.py): `task_datasets/humaneval.py`
- [Overruling loader](https://github.com/confidence-routing/confidence-tool-routing/blob/main/task_datasets/overruling.py): `task_datasets/overruling.py`

### Evaluation core

- [TaskRecord schema](https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/models.py): `eval_harness/models.py`
- [Task grading](https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/grading.py): `eval_harness/grading.py`
- [Evaluation metrics](https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/metrics.py): `eval_harness/metrics.py`
- [Cost estimation](https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/costs.py): `eval_harness/costs.py`
- [Run reports](https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/report.py): `eval_harness/report.py`

### Confidence and routing

- [Confidence estimators](https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/confidence.py): `eval_harness/confidence.py`
- [Routing and sweeps](https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/router.py): `eval_harness/router.py`

### Tools and persistence

- [Calculator and runner](https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/tools.py): `eval_harness/tools.py`
- [JSONL run logger](https://github.com/confidence-routing/confidence-tool-routing/blob/main/eval_harness/logger.py): `eval_harness/logger.py`

### Other components

- **Experimenter**
- **Dataset sources**

## Connections

- Experimenter → Dataset registry: selects dataset (evidence: `README.md`)
- Dataset registry → CoQA loader: loads via registry (evidence: `task_datasets/base.py`)
- Dataset registry → GSM8K loader: loads via registry (evidence: `task_datasets/base.py`)
- Dataset registry → Headlines loader: loads via registry (evidence: `task_datasets/base.py`)
- Dataset registry → HumanEval loader: loads via registry (evidence: `task_datasets/base.py`)
- Dataset registry → Overruling loader: loads via registry (evidence: `task_datasets/base.py`)
- Dataset sources → Headlines loader: provides rows (evidence: `task_datasets/headlines.py`)
- Dataset sources → Overruling loader: provides rows (evidence: `task_datasets/overruling.py`)
- Dataset sources → HumanEval loader: provides benchmark (evidence: `task_datasets/humaneval.py`)
- Dataset registry → TaskRecord schema: supplies task inputs
- TaskRecord schema → Task grading: grades answers (evidence: `eval_harness/grading.py`)
- Task grading → Calculator and runner: runs code tests (evidence: `eval_harness/grading.py`)
- TaskRecord schema → Evaluation metrics: measures records (evidence: `eval_harness/metrics.py`)
- Evaluation metrics → Cost estimation: estimates costs (evidence: `eval_harness/metrics.py`)
- TaskRecord schema → JSONL run logger: serializes records (evidence: `eval_harness/logger.py`)
- TaskRecord schema → Confidence estimators: stores confidence (evidence: `eval_harness/confidence.py`)
- Confidence estimators → Routing and sweeps: supplies confidence (evidence: `README.md`)
- TaskRecord schema → Routing and sweeps: sets routing fields (evidence: `eval_harness/router.py`)
- Routing and sweeps → Evaluation metrics: grades sweeps (evidence: `eval_harness/router.py`)
- TaskRecord schema → Run reports: summarizes records (evidence: `eval_harness/report.py`)
- Evaluation metrics → Run reports: aggregates metrics (evidence: `eval_harness/report.py`)
