# cubedevinc/semantic-layer-benchmark architecture

> Architecture diagram of the GitHub repository cubedevinc/semantic-layer-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cubedevinc/semantic-layer-benchmark
- Repository: https://github.com/cubedevinc/semantic-layer-benchmark

## Overview

This repository is a reproducible text-to-SQL benchmark comparing model accuracy with raw schema context versus an added hand-authored semantic layer. Its main workflow executes reference SQL, runs model-generated SQL, judges candidate results against reference results, and reports accuracy and paired statistics. It also supports schema retrieval and generating candidate benchmark questions. The graph follows the documented workflow and sampled script calls; unsampled internal module behavior is represented only where those scripts or the README identify it.

Main entry points: [`scripts/run_claude.py`](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/run_claude.py) (Claude runner), [`scripts/run_codex.py`](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/run_codex.py) (Codex runner), [`benchmarks/text_to_sql/clickhouse_client.py`](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/benchmarks/text_to_sql/clickhouse_client.py) (ClickHouse client), [`benchmarks/text_to_sql/benchmark_data.py`](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/benchmarks/text_to_sql/benchmark_data.py) (Benchmark loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_authoring["Benchmark authoring"]
  node_question_generator["Question generator"]
  node_schema_fetcher["Schema fetcher<br/>[fetch_schema.py]"]
  node_schema_fixture["Retail schema<br/>[retail_schema.sql]"]
  node_semantic_guide["Semantic guide<br/>[semantic_layer.md]"]
end

subgraph group_execution["Query execution"]
  node_reference_runner["Reference runner<br/>[run_reference.py]"]
  node_claude_runner["Claude runner<br/>[run_claude.py]"]
  node_codex_runner["Codex runner<br/>[run_codex.py]"]
  node_claude_adapters["Claude adapters"]
  node_codex_adapter["Codex adapter<br/>[codex.py]"]
end

subgraph group_evaluation["Scoring and reporting"]
  node_score_runner["Scoring runner<br/>[score_run.py]"]
  node_judge["Result judge<br/>[scorer.py]"]
  node_reporter["Report generator<br/>[report_run.py]"]
end

subgraph group_infrastructure["Shared infrastructure"]
  node_clickhouse_client["ClickHouse client"]
  node_run_storage[("Run artifacts")]
  node_benchmark_data["Benchmark loader<br/>[benchmark_data.py]"]
  node_run_paths["Run paths<br/>[paths.py]"]
end

node_analyst(("Benchmark operator"))
node_model_services{{"LLM services"}}
node_clickhouse[("ClickHouse")]

node_analyst -->|"starts run"| node_reference_runner
node_analyst -->|"starts run"| node_claude_runner
node_analyst -->|"starts run"| node_codex_runner
node_analyst -->|"starts scoring"| node_score_runner
node_analyst -->|"requests report"| node_reporter
node_analyst -->|"generates candidates"| node_question_generator
node_schema_fetcher -->|"uses client"| node_clickhouse_client
node_schema_fetcher -->|"writes DDL"| node_schema_fixture
node_question_generator -->|"validates SQL"| node_clickhouse_client
node_question_generator -->|"reads DDL"| node_schema_fixture
node_question_generator -->|"reads guide"| node_semantic_guide
node_reference_runner -->|"loads questions"| node_benchmark_data
node_reference_runner -->|"executes SQL"| node_clickhouse_client
node_reference_runner -->|"writes references"| node_run_paths
node_claude_runner -->|"loads questions"| node_benchmark_data
node_claude_runner -->|"dispatches prompt"| node_claude_adapters
node_claude_runner -->|"reads DDL"| node_schema_fixture
node_claude_runner -->|"optionally reads"| node_semantic_guide
node_claude_adapters -->|"requests SQL"| node_model_services
node_claude_runner -->|"executes SQL"| node_clickhouse_client
node_claude_runner -->|"writes outputs"| node_run_paths
node_codex_runner -->|"loads questions"| node_benchmark_data
node_codex_runner -->|"dispatches prompt"| node_codex_adapter
node_codex_runner -->|"reads DDL"| node_schema_fixture
node_codex_runner -->|"optionally reads"| node_semantic_guide
node_codex_adapter -->|"requests SQL"| node_model_services
node_codex_runner -->|"executes SQL"| node_clickhouse_client
node_codex_runner -->|"writes outputs"| node_run_paths
node_clickhouse_client -->|"queries database"| node_clickhouse
node_score_runner -->|"loads questions"| node_benchmark_data
node_score_runner -->|"reads and writes"| node_run_paths
node_score_runner -->|"scores pairs"| node_judge
node_judge -->|"requests verdict"| node_model_services
node_reporter -->|"loads questions"| node_benchmark_data
node_reporter -->|"reads score files"| node_run_storage

click node_question_generator "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/generate_questions.py"
click node_schema_fetcher "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/fetch_schema.py"
click node_schema_fixture "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/fixtures/retail_schema.sql"
click node_semantic_guide "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/fixtures/semantic_layer.md"
click node_reference_runner "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/run_reference.py"
click node_claude_runner "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/run_claude.py"
click node_codex_runner "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/run_codex.py"
click node_claude_adapters "https://github.com/cubedevinc/semantic-layer-benchmark/tree/master/benchmarks/text_to_sql/runners"
click node_codex_adapter "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/benchmarks/text_to_sql/runners/codex.py"
click node_clickhouse_client "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/benchmarks/text_to_sql/clickhouse_client.py"
click node_run_storage "https://github.com/cubedevinc/semantic-layer-benchmark/tree/master/published-runs"
click node_score_runner "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/score_run.py"
click node_judge "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/benchmarks/text_to_sql/scorer.py"
click node_reporter "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/report_run.py"
click node_benchmark_data "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/benchmarks/text_to_sql/benchmark_data.py"
click node_run_paths "https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/benchmarks/text_to_sql/paths.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_question_generator,node_schema_fetcher,node_schema_fixture,node_semantic_guide toneBlue
class node_reference_runner,node_claude_runner,node_codex_runner,node_claude_adapters,node_codex_adapter,node_clickhouse toneAmber
class node_score_runner,node_judge,node_reporter toneMint
class node_clickhouse_client,node_run_storage,node_benchmark_data,node_run_paths toneRose
class node_analyst,node_model_services toneIndigo
```

## Components

### Benchmark authoring

- [Question generator](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/generate_questions.py): `scripts/generate_questions.py`
- [Schema fetcher](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/fetch_schema.py): `scripts/fetch_schema.py`
- [Retail schema](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/fixtures/retail_schema.sql): `fixtures/retail_schema.sql`
- [Semantic guide](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/fixtures/semantic_layer.md): `fixtures/semantic_layer.md`

### Query execution

- [Reference runner](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/run_reference.py): `scripts/run_reference.py`
- [Claude runner](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/run_claude.py): `scripts/run_claude.py`
- [Codex runner](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/run_codex.py): `scripts/run_codex.py`
- [Claude adapters](https://github.com/cubedevinc/semantic-layer-benchmark/tree/master/benchmarks/text_to_sql/runners): `benchmarks/text_to_sql/runners`
- [Codex adapter](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/benchmarks/text_to_sql/runners/codex.py): `benchmarks/text_to_sql/runners/codex.py`

### Scoring and reporting

- [Scoring runner](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/score_run.py): `scripts/score_run.py`
- [Result judge](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/benchmarks/text_to_sql/scorer.py): `benchmarks/text_to_sql/scorer.py`
- [Report generator](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/scripts/report_run.py): `scripts/report_run.py`

### Shared infrastructure

- [ClickHouse client](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/benchmarks/text_to_sql/clickhouse_client.py): `benchmarks/text_to_sql/clickhouse_client.py`
- [Run artifacts](https://github.com/cubedevinc/semantic-layer-benchmark/tree/master/published-runs): `published-runs`
- [Benchmark loader](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/benchmarks/text_to_sql/benchmark_data.py): `benchmarks/text_to_sql/benchmark_data.py`
- [Run paths](https://github.com/cubedevinc/semantic-layer-benchmark/blob/master/benchmarks/text_to_sql/paths.py): `benchmarks/text_to_sql/paths.py`

### Other components

- **Benchmark operator**
- **LLM services**
- **ClickHouse**

## Connections

- Benchmark operator → Reference runner: starts run (evidence: `README.md`)
- Benchmark operator → Claude runner: starts run (evidence: `README.md`)
- Benchmark operator → Codex runner: starts run (evidence: `README.md`)
- Benchmark operator → Scoring runner: starts scoring (evidence: `README.md`)
- Benchmark operator → Report generator: requests report (evidence: `README.md`)
- Benchmark operator → Question generator: generates candidates (evidence: `scripts/generate_questions.py`)
- Schema fetcher → ClickHouse client: uses client (evidence: `scripts/fetch_schema.py`)
- Schema fetcher → Retail schema: writes DDL (evidence: `scripts/fetch_schema.py`)
- Question generator → ClickHouse client: validates SQL (evidence: `scripts/generate_questions.py`)
- Question generator → Retail schema: reads DDL (evidence: `scripts/generate_questions.py`)
- Question generator → Semantic guide: reads guide (evidence: `scripts/generate_questions.py`)
- Reference runner → Benchmark loader: loads questions (evidence: `scripts/run_reference.py`)
- Reference runner → ClickHouse client: executes SQL (evidence: `scripts/run_reference.py`)
- Reference runner → Run paths: writes references (evidence: `scripts/run_reference.py`)
- Claude runner → Benchmark loader: loads questions (evidence: `scripts/run_claude.py`)
- Claude runner → Claude adapters: dispatches prompt (evidence: `scripts/run_claude.py`)
- Claude runner → Retail schema: reads DDL (evidence: `scripts/run_claude.py`)
- Claude runner → Semantic guide: optionally reads (evidence: `scripts/run_claude.py`)
- Claude adapters → LLM services: requests SQL (evidence: `README.md`)
- Claude runner → ClickHouse client: executes SQL (evidence: `scripts/run_claude.py`)
- Claude runner → Run paths: writes outputs (evidence: `scripts/run_claude.py`)
- Codex runner → Benchmark loader: loads questions (evidence: `scripts/run_codex.py`)
- Codex runner → Codex adapter: dispatches prompt (evidence: `scripts/run_codex.py`)
- Codex runner → Retail schema: reads DDL (evidence: `scripts/run_codex.py`)
- Codex runner → Semantic guide: optionally reads (evidence: `scripts/run_codex.py`)
- Codex adapter → LLM services: requests SQL (evidence: `README.md`)
- Codex runner → ClickHouse client: executes SQL (evidence: `scripts/run_codex.py`)
- Codex runner → Run paths: writes outputs (evidence: `scripts/run_codex.py`)
- ClickHouse client → ClickHouse: queries database (evidence: `README.md`)
- Scoring runner → Benchmark loader: loads questions (evidence: `scripts/score_run.py`)
- Scoring runner → Run paths: reads and writes (evidence: `scripts/score_run.py`)
- Scoring runner → Result judge: scores pairs (evidence: `scripts/score_run.py`)
- Result judge → LLM services: requests verdict (evidence: `README.md`)
- Report generator → Benchmark loader: loads questions (evidence: `scripts/report_run.py`)
- Report generator → Run artifacts: reads score files (evidence: `README.md`)
