How cubedevinc/semantic-layer-benchmark works
This repository is a reproducible text-to-SQL benchmark comparing model accuracy with raw schema context versus an added hand-authored semantic layer. Its main workflow executes reference SQL, runs model-generated SQL, judges candidate results against reference results, and reports accuracy and paired statistics. It also supports schema retrieval and generating candidate benchmark questions. The graph follows the documented workflow and sampled script calls; unsampled internal module behavior is represented only where those scripts or the README identify it.
Main entry points: run_claude.py Claude runner · run_codex.py Codex runner · clickhouse_client.py ClickHouse client · benchmark_data.py Benchmark loader
Components 19
Benchmark authoring
- Question generator
scripts/generate_questions.py - Schema fetcher
scripts/fetch_schema.py - Retail schema
fixtures/retail_schema.sql - Semantic guide
fixtures/semantic_layer.md
Query execution
- Reference runner
scripts/run_reference.py - Claude runner
scripts/run_claude.py - Codex runner
scripts/run_codex.py - Claude adapters
benchmarks/text_to_sql/runners - Codex adapter
benchmarks/text_to_sql/runners/codex.py
Scoring and reporting
- Scoring runner
scripts/score_run.py - Result judge
benchmarks/text_to_sql/scorer.py - Report generator
scripts/report_run.py
Shared infrastructure
- ClickHouse client
benchmarks/text_to_sql/clickhouse_client.py - Run artifacts
published-runs - Benchmark loader
benchmarks/text_to_sql/benchmark_data.py - Run paths
benchmarks/text_to_sql/paths.py
Other components
- Benchmark operator
- LLM services
- ClickHouse
Connections 35
- Benchmark operator → Reference runner: starts run
README.md - Benchmark operator → Claude runner: starts run
README.md - Benchmark operator → Codex runner: starts run
README.md - Benchmark operator → Scoring runner: starts scoring
README.md - Benchmark operator → Report generator: requests report
README.md - Benchmark operator → Question generator: generates candidates
scripts/generate_questions.py - Schema fetcher → ClickHouse client: uses client
scripts/fetch_schema.py - Schema fetcher → Retail schema: writes DDL
scripts/fetch_schema.py - Question generator → ClickHouse client: validates SQL
scripts/generate_questions.py - Question generator → Retail schema: reads DDL
scripts/generate_questions.py - Question generator → Semantic guide: reads guide
scripts/generate_questions.py - Reference runner → Benchmark loader: loads questions
scripts/run_reference.py - Reference runner → ClickHouse client: executes SQL
scripts/run_reference.py - Reference runner → Run paths: writes references
scripts/run_reference.py - Claude runner → Benchmark loader: loads questions
scripts/run_claude.py - Claude runner → Claude adapters: dispatches prompt
scripts/run_claude.py - Claude runner → Retail schema: reads DDL
scripts/run_claude.py - Claude runner → Semantic guide: optionally reads
scripts/run_claude.py - Claude adapters → LLM services: requests SQL
README.md - Claude runner → ClickHouse client: executes SQL
scripts/run_claude.py - Claude runner → Run paths: writes outputs
scripts/run_claude.py - Codex runner → Benchmark loader: loads questions
scripts/run_codex.py - Codex runner → Codex adapter: dispatches prompt
scripts/run_codex.py - Codex runner → Retail schema: reads DDL
scripts/run_codex.py - Codex runner → Semantic guide: optionally reads
scripts/run_codex.py - Codex adapter → LLM services: requests SQL
README.md - Codex runner → ClickHouse client: executes SQL
scripts/run_codex.py - Codex runner → Run paths: writes outputs
scripts/run_codex.py - ClickHouse client → ClickHouse: queries database
README.md - Scoring runner → Benchmark loader: loads questions
scripts/score_run.py - Scoring runner → Run paths: reads and writes
scripts/score_run.py - Scoring runner → Result judge: scores pairs
scripts/score_run.py - Result judge → LLM services: requests verdict
README.md - Report generator → Benchmark loader: loads questions
scripts/report_run.py - Report generator → Run artifacts: reads score files
README.md