How shreyshringare/querylens works
QueryLens is a database-agnostic Text-to-SQL data preparation pipeline. Its principal workflow adapts Spider, BIRD, or Synthetic records into a common normalized record, then validates, deduplicates, splits, and reports on the dataset. The Phase 2 branch parses DDL and SQL into structured representations and can cross-check SQL references against a schema. README-described schema linking is a planned capability, not an implemented component visible in the tree. Relationships follow README documentation and indexed imports; unsampled implementation details are not inferred.
Main entry points: data_pipeline/models.py Normalized record · schema/models.py Database schema · adapt.py Adapter dispatch · normalizer.py Query normalizer
Components 24
Dataset ingestion
- Spider adapter
src/data_pipeline/adapters/spider.py - BIRD adapter
src/data_pipeline/adapters/bird.py - Synthetic adapter
src/data_pipeline/adapters/synthetic.py - Adapter dispatch
src/data_pipeline/adapt.py - Record normalization
src/data_pipeline/pipeline.py - Query normalizer
src/data_pipeline/normalizer.py - Dialect mapping
src/data_pipeline/dialects.py - Normalized record
src/data_pipeline/models.py
Dataset curation
- Record validation
src/data_pipeline/validation.py - Deduplication
src/data_pipeline/dedup.py - Dataset splitting
src/data_pipeline/split.py - Quality reporting
src/data_pipeline/quality.py - Dataset statistics
src/data_pipeline/stats.py - Normalized dataset
data/processed/normalized.jsonl - Train validation test
data/processed/train.jsonl
Schema understanding
- DDL schema parser
src/schema/parser.py - Database schema
src/schema/models.py - Schema validation
src/schema/validator.py - Schema relationships
src/schema/relationship_graph.py - Schema serializer
src/schema/serializer.py
SQL representation
- SQL parser
src/sql/parser.py - SQL structure
src/sql/models.py - Alias resolver
src/sql/aliases.py
Other components
- Schema linking
Connections 26
- Record normalization → Adapter dispatch: adapts through
src/data_pipeline/pipeline.py - Adapter dispatch → Spider adapter: dispatches to
- Adapter dispatch → BIRD adapter: dispatches to
- Adapter dispatch → Synthetic adapter: dispatches to
- Record normalization → Query normalizer: normalizes with
src/data_pipeline/pipeline.py - Query normalizer → Dialect mapping: normalizes dialect
src/data_pipeline/normalizer.py - Query normalizer → Normalized record: produces
src/data_pipeline/normalizer.py - Normalized record → Record validation: validated by
README.md - Normalized record → Deduplication: deduplicated by
README.md - Normalized record → Dataset splitting: split by
README.md - Dataset splitting → Train validation test: writes splits
README.md - Normalized record → Quality reporting: quality checked
src/data_pipeline/quality.py - Quality reporting → Dataset statistics: uses statistics
src/data_pipeline/quality.py - Normalized record → Dataset statistics: summarized by
README.md - Normalized record → Normalized dataset: stored as
README.md - Normalized dataset → DDL schema parser: provides DDL
README.md - Normalized dataset → SQL parser: provides SQL
README.md - DDL schema parser → Database schema: builds
src/schema/parser.py - Database schema → Schema serializer: serialized by
src/schema/serializer.py - Database schema → Schema validation: validated by
src/schema/validator.py - Database schema → Schema relationships: models relations
src/schema/relationship_graph.py - SQL parser → SQL structure: builds
src/sql/parser.py - SQL parser → Alias resolver: resolves aliases
src/sql/parser.py - SQL structure → Schema validation: cross-checked by
src/schema/validator.py - Database schema → Schema linking: supports planned
README.md - SQL structure → Schema linking: supports planned
README.md