How confidence-routing/confidence-tool-routing works
This repository is a measurement and evaluation harness for confidence-based tool routing, with dataset loaders, task grading, confidence estimators, routing analysis, and run reporting. The intended task-to-result flow is documented, but the README explicitly says the real model/tool runner is not yet implemented; the graph therefore does not invent that orchestration. Dataset-specific loaders and several evaluation relationships are included from the source index and excerpts, while unsampled implementation details are not inferred.
Main entry points: base.py Dataset registry · models.py TaskRecord schema · metrics.py Evaluation metrics · router.py Routing and sweeps
Components 17
Task datasets
- Dataset registry
task_datasets/base.py - CoQA loader
task_datasets/coqa.py - GSM8K loader
task_datasets/gsm8k.py - Headlines loader
task_datasets/headlines.py - HumanEval loader
task_datasets/humaneval.py - Overruling loader
task_datasets/overruling.py
Evaluation core
- TaskRecord schema
eval_harness/models.py - Task grading
eval_harness/grading.py - Evaluation metrics
eval_harness/metrics.py - Cost estimation
eval_harness/costs.py - Run reports
eval_harness/report.py
Confidence and routing
- Confidence estimators
eval_harness/confidence.py - Routing and sweeps
eval_harness/router.py
Tools and persistence
- Calculator and runner
eval_harness/tools.py - JSONL run logger
eval_harness/logger.py
Other components
- Experimenter
- Dataset sources
Connections 21
- Experimenter → Dataset registry: selects dataset
README.md - Dataset registry → CoQA loader: loads via registry
task_datasets/base.py - Dataset registry → GSM8K loader: loads via registry
task_datasets/base.py - Dataset registry → Headlines loader: loads via registry
task_datasets/base.py - Dataset registry → HumanEval loader: loads via registry
task_datasets/base.py - Dataset registry → Overruling loader: loads via registry
task_datasets/base.py - Dataset sources → Headlines loader: provides rows
task_datasets/headlines.py - Dataset sources → Overruling loader: provides rows
task_datasets/overruling.py - Dataset sources → HumanEval loader: provides benchmark
task_datasets/humaneval.py - Dataset registry → TaskRecord schema: supplies task inputs
- TaskRecord schema → Task grading: grades answers
eval_harness/grading.py - Task grading → Calculator and runner: runs code tests
eval_harness/grading.py - TaskRecord schema → Evaluation metrics: measures records
eval_harness/metrics.py - Evaluation metrics → Cost estimation: estimates costs
eval_harness/metrics.py - TaskRecord schema → JSONL run logger: serializes records
eval_harness/logger.py - TaskRecord schema → Confidence estimators: stores confidence
eval_harness/confidence.py - Confidence estimators → Routing and sweeps: supplies confidence
README.md - TaskRecord schema → Routing and sweeps: sets routing fields
eval_harness/router.py - Routing and sweeps → Evaluation metrics: grades sweeps
eval_harness/router.py - TaskRecord schema → Run reports: summarizes records
eval_harness/report.py - Evaluation metrics → Run reports: aggregates metrics
eval_harness/report.py