# shreyshringare/querylens architecture

> Architecture diagram of the GitHub repository shreyshringare/querylens, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shreyshringare/querylens
- Repository: https://github.com/shreyshringare/querylens

## Overview

QueryLens is a database-agnostic Text-to-SQL data preparation pipeline. Its principal workflow adapts Spider, BIRD, or Synthetic records into a common normalized record, then validates, deduplicates, splits, and reports on the dataset. The Phase 2 branch parses DDL and SQL into structured representations and can cross-check SQL references against a schema. README-described schema linking is a planned capability, not an implemented component visible in the tree. Relationships follow README documentation and indexed imports; unsampled implementation details are not inferred.

Main entry points: [`src/data_pipeline/models.py`](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/models.py) (Normalized record), [`src/schema/models.py`](https://github.com/shreyshringare/querylens/blob/main/src/schema/models.py) (Database schema), [`src/data_pipeline/adapt.py`](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/adapt.py) (Adapter dispatch), [`src/data_pipeline/normalizer.py`](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/normalizer.py) (Query normalizer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Dataset ingestion"]
  node_spider["Spider adapter<br/>[spider.py]"]
  node_bird["BIRD adapter<br/>[bird.py]"]
  node_synthetic["Synthetic adapter<br/>[synthetic.py]"]
  node_adapter_dispatch["Adapter dispatch<br/>[adapt.py]"]
  node_record_pipeline["Record normalization<br/>[pipeline.py]"]
  node_normalizer["Query normalizer<br/>[normalizer.py]"]
  node_dialects["Dialect mapping<br/>[dialects.py]"]
  node_normalized_record["Normalized record<br/>[models.py]"]
end

subgraph group_curation["Dataset curation"]
  node_validation["Record validation<br/>[validation.py]"]
  node_dedup["Deduplication<br/>[dedup.py]"]
  node_splitting["Dataset splitting<br/>[split.py]"]
  node_quality["Quality reporting<br/>[quality.py]"]
  node_statistics["Dataset statistics<br/>[stats.py]"]
  node_normalized_data["Normalized dataset<br/>[normalized.jsonl]"]
  node_split_data["Train validation test<br/>[train.jsonl]"]
end

subgraph group_schema["Schema understanding"]
  node_schema_parser["DDL schema parser<br/>[parser.py]"]
  node_schema_model["Database schema<br/>[models.py]"]
  node_schema_validator["Schema validation<br/>[validator.py]"]
  node_schema_relations["Schema relationships"]
  node_schema_serializer["Schema serializer<br/>[serializer.py]"]
end

subgraph group_sql["SQL representation"]
  node_sql_parser["SQL parser<br/>[parser.py]"]
  node_sql_model["SQL structure<br/>[models.py]"]
  node_alias_resolver["Alias resolver<br/>[aliases.py]"]
end

node_schema_linking["Schema linking"]

node_record_pipeline -->|"adapts through"| node_adapter_dispatch
node_adapter_dispatch -->|"dispatches to"| node_spider
node_adapter_dispatch -->|"dispatches to"| node_bird
node_adapter_dispatch -->|"dispatches to"| node_synthetic
node_record_pipeline -->|"normalizes with"| node_normalizer
node_normalizer -->|"normalizes dialect"| node_dialects
node_normalizer -->|"produces"| node_normalized_record
node_normalized_record -->|"validated by"| node_validation
node_normalized_record -->|"deduplicated by"| node_dedup
node_normalized_record -->|"split by"| node_splitting
node_splitting -->|"writes splits"| node_split_data
node_normalized_record -->|"quality checked"| node_quality
node_quality -->|"uses statistics"| node_statistics
node_normalized_record -->|"summarized by"| node_statistics
node_normalized_record -->|"stored as"| node_normalized_data
node_normalized_data -->|"provides DDL"| node_schema_parser
node_normalized_data -->|"provides SQL"| node_sql_parser
node_schema_parser -->|"builds"| node_schema_model
node_schema_model -->|"serialized by"| node_schema_serializer
node_schema_model -->|"validated by"| node_schema_validator
node_schema_model -->|"models relations"| node_schema_relations
node_sql_parser -->|"builds"| node_sql_model
node_sql_parser -->|"resolves aliases"| node_alias_resolver
node_sql_model -->|"cross-checked by"| node_schema_validator
node_schema_model -.->|"supports planned"| node_schema_linking
node_sql_model -.->|"supports planned"| node_schema_linking

click node_spider "https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/adapters/spider.py"
click node_bird "https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/adapters/bird.py"
click node_synthetic "https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/adapters/synthetic.py"
click node_adapter_dispatch "https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/adapt.py"
click node_record_pipeline "https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/pipeline.py"
click node_normalizer "https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/normalizer.py"
click node_dialects "https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/dialects.py"
click node_normalized_record "https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/models.py"
click node_validation "https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/validation.py"
click node_dedup "https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/dedup.py"
click node_splitting "https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/split.py"
click node_quality "https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/quality.py"
click node_statistics "https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/stats.py"
click node_normalized_data "https://github.com/shreyshringare/querylens/blob/main/data/processed/normalized.jsonl"
click node_split_data "https://github.com/shreyshringare/querylens/blob/main/data/processed/train.jsonl"
click node_schema_parser "https://github.com/shreyshringare/querylens/blob/main/src/schema/parser.py"
click node_schema_model "https://github.com/shreyshringare/querylens/blob/main/src/schema/models.py"
click node_schema_validator "https://github.com/shreyshringare/querylens/blob/main/src/schema/validator.py"
click node_schema_relations "https://github.com/shreyshringare/querylens/blob/main/src/schema/relationship_graph.py"
click node_schema_serializer "https://github.com/shreyshringare/querylens/blob/main/src/schema/serializer.py"
click node_sql_parser "https://github.com/shreyshringare/querylens/blob/main/src/sql/parser.py"
click node_sql_model "https://github.com/shreyshringare/querylens/blob/main/src/sql/models.py"
click node_alias_resolver "https://github.com/shreyshringare/querylens/blob/main/src/sql/aliases.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_spider,node_bird,node_synthetic,node_adapter_dispatch,node_record_pipeline,node_normalizer,node_dialects,node_normalized_record toneBlue
class node_validation,node_dedup,node_splitting,node_quality,node_statistics,node_normalized_data,node_split_data toneAmber
class node_schema_parser,node_schema_model,node_schema_validator,node_schema_relations,node_schema_serializer toneMint
class node_sql_parser,node_sql_model,node_alias_resolver toneRose
class node_schema_linking toneIndigo
```

## Components

### Dataset ingestion

- [Spider adapter](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/adapters/spider.py): `src/data_pipeline/adapters/spider.py`
- [BIRD adapter](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/adapters/bird.py): `src/data_pipeline/adapters/bird.py`
- [Synthetic adapter](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/adapters/synthetic.py): `src/data_pipeline/adapters/synthetic.py`
- [Adapter dispatch](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/adapt.py): `src/data_pipeline/adapt.py`
- [Record normalization](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/pipeline.py): `src/data_pipeline/pipeline.py`
- [Query normalizer](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/normalizer.py): `src/data_pipeline/normalizer.py`
- [Dialect mapping](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/dialects.py): `src/data_pipeline/dialects.py`
- [Normalized record](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/models.py): `src/data_pipeline/models.py`

### Dataset curation

- [Record validation](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/validation.py): `src/data_pipeline/validation.py`
- [Deduplication](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/dedup.py): `src/data_pipeline/dedup.py`
- [Dataset splitting](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/split.py): `src/data_pipeline/split.py`
- [Quality reporting](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/quality.py): `src/data_pipeline/quality.py`
- [Dataset statistics](https://github.com/shreyshringare/querylens/blob/main/src/data_pipeline/stats.py): `src/data_pipeline/stats.py`
- [Normalized dataset](https://github.com/shreyshringare/querylens/blob/main/data/processed/normalized.jsonl): `data/processed/normalized.jsonl`
- [Train validation test](https://github.com/shreyshringare/querylens/blob/main/data/processed/train.jsonl): `data/processed/train.jsonl`

### Schema understanding

- [DDL schema parser](https://github.com/shreyshringare/querylens/blob/main/src/schema/parser.py): `src/schema/parser.py`
- [Database schema](https://github.com/shreyshringare/querylens/blob/main/src/schema/models.py): `src/schema/models.py`
- [Schema validation](https://github.com/shreyshringare/querylens/blob/main/src/schema/validator.py): `src/schema/validator.py`
- [Schema relationships](https://github.com/shreyshringare/querylens/blob/main/src/schema/relationship_graph.py): `src/schema/relationship_graph.py`
- [Schema serializer](https://github.com/shreyshringare/querylens/blob/main/src/schema/serializer.py): `src/schema/serializer.py`

### SQL representation

- [SQL parser](https://github.com/shreyshringare/querylens/blob/main/src/sql/parser.py): `src/sql/parser.py`
- [SQL structure](https://github.com/shreyshringare/querylens/blob/main/src/sql/models.py): `src/sql/models.py`
- [Alias resolver](https://github.com/shreyshringare/querylens/blob/main/src/sql/aliases.py): `src/sql/aliases.py`

### Other components

- **Schema linking**

## Connections

- Record normalization → Adapter dispatch: adapts through (evidence: `src/data_pipeline/pipeline.py`)
- Adapter dispatch → Spider adapter: dispatches to
- Adapter dispatch → BIRD adapter: dispatches to
- Adapter dispatch → Synthetic adapter: dispatches to
- Record normalization → Query normalizer: normalizes with (evidence: `src/data_pipeline/pipeline.py`)
- Query normalizer → Dialect mapping: normalizes dialect (evidence: `src/data_pipeline/normalizer.py`)
- Query normalizer → Normalized record: produces (evidence: `src/data_pipeline/normalizer.py`)
- Normalized record → Record validation: validated by (evidence: `README.md`)
- Normalized record → Deduplication: deduplicated by (evidence: `README.md`)
- Normalized record → Dataset splitting: split by (evidence: `README.md`)
- Dataset splitting → Train validation test: writes splits (evidence: `README.md`)
- Normalized record → Quality reporting: quality checked (evidence: `src/data_pipeline/quality.py`)
- Quality reporting → Dataset statistics: uses statistics (evidence: `src/data_pipeline/quality.py`)
- Normalized record → Dataset statistics: summarized by (evidence: `README.md`)
- Normalized record → Normalized dataset: stored as (evidence: `README.md`)
- Normalized dataset → DDL schema parser: provides DDL (evidence: `README.md`)
- Normalized dataset → SQL parser: provides SQL (evidence: `README.md`)
- DDL schema parser → Database schema: builds (evidence: `src/schema/parser.py`)
- Database schema → Schema serializer: serialized by (evidence: `src/schema/serializer.py`)
- Database schema → Schema validation: validated by (evidence: `src/schema/validator.py`)
- Database schema → Schema relationships: models relations (evidence: `src/schema/relationship_graph.py`)
- SQL parser → SQL structure: builds (evidence: `src/sql/parser.py`)
- SQL parser → Alias resolver: resolves aliases (evidence: `src/sql/parser.py`)
- SQL structure → Schema validation: cross-checked by (evidence: `src/schema/validator.py`)
- Database schema → Schema linking: supports planned (evidence: `README.md`)
- SQL structure → Schema linking: supports planned (evidence: `README.md`)
