# tobymao/sqlglot architecture

> Architecture diagram of the GitHub repository tobymao/sqlglot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tobymao/sqlglot
- Repository: https://github.com/tobymao/sqlglot

## Overview

1. **Project shape**
   - This is a Python library centered on a single core package, `sqlglot`, with optional compiled acceleration in `sqlglotc`.
   - It is not an application with servers or jobs; the architecture is a reusable SQL processing engine consumed as a library and CLI entry point.

2. **Primary pipeline**
   - The main data flow is: SQL text in → tokenization → parsing → expression tree (AST) → optional optimization/analysis → SQL generation or execution.
   - Most features in the repo are built around that pipeline, with dialect-specific behavior injected at parse and generation time.

3. **Parsing subsystem**
   - `sqlglot/tokenizer_core.py`, `tokens.py`, and `parsers/` implement lexical analysis and syntax parsing.
   - `parser.py` and dialect-specific parser modules translate dialect-specific SQL into a shared internal AST.
   - The parser is designed to accept a broad “SQLGlot dialect” plus named source dialects such as Spark, BigQuery, Snowflake, Trino, and others.

4. **AST / expression model**
   - `sqlglot/expressions/` defines the internal tree of SQL nodes: queries, DDL/DML, functions, datatypes, temporal expressions, JSON, constraints, etc.
   - This AST is the central boundary between all subsystems: parser produces it, optimizer rewrites it, generator renders it, lineage/diff inspect it, and executor interprets it.

5. **Dialect layer**
   - `sqlglot/dialects/` contains one dialect class per supported SQL flavor.
   - A dialect bundles a tokenizer, parser behavior, and generator behavior so the same AST can be read from one dialect and emitted in another.
   - This is the key extensibility boundary; plugin dialects can be registered externally through entry points.

6. **SQL generation / transpilation**
   - `generator.py` and `generators/` turn AST nodes back into SQL text.
   - Dialect-specific generator modules control quoting, type names, function mappings, syntax differences, and formatting details.
   - Transpilation is simply AST generation followed by rendering into a target dialect.

7. **Optimizer subsystem**
   - `sqlglot/optimizer/` contains rule-based query rewrites such as qualification, normalization, predicate pushdown, projection pushdown, subquery/CTE elimination, join optimization, and simplification.
   - The optimizer takes an AST plus optional schema metadata and returns a transformed canonical AST.
   - This is the main semantic-analysis layer and is used when the library needs more than syntax translation.

8. **Analysis and metadata utilities**
   - `lineage.py`, `diff.py`, `schema.py`, and `transforms.py` provide higher-level analysis on top of the AST.
   - `lineage` traces how output columns depend on input tables/expressions.
   - `diff` computes semantic differences between SQL trees.
   - `schema` provides table/column type information needed by optimizer rules and type-sensitive transpilation.

9. **Execution engine**
   - `sqlglot/executor/` is a lightweight Python SQL execution engine for running parsed SQL against in-memory table data.
   - It is intended for correctness, testing, and educational use rather than performance.
   - The executor consumes the same AST produced by the parser, which keeps execution aligned with the library’s parsing and rewrite semantics.

10. **Supporting utilities**
   - `helper.py`, `serde.py`, `jsonpath.py`, `trie.py`, `time.py`, and `typing/` support common concerns like serialization, path parsing, keyword lookup, temporal handling, and dialect-specific type logic.
   - `errors.py` defines the structured error model used by parser and transpiler failures.

11. **Optional performance path**
   - `sqlglotc/` packages a compiled variant via mypyc, giving a faster runtime path without changing the public architecture.
   - The repository therefore has two execution modes: pure Python and compiled extension-backed, both using the same library surface.

12. **Testing architecture**
   - `tests/` is extensive and organized by subsystem: parser, generator, optimizer, executor, lineage, diff, schema, serde, tokens, transforms, and per-dialect behavior.
   - Fixtures under `tests/fixtures/` drive deterministic coverage for optimizer rewrites, JSONPath, and large SQL corpora.
   - There is also a separate `sqlglot-integration-tests` area and CI workflows for sync and verification.

13. **Tooling and release flow**
   - Packaging is Python-first (`pyproject.toml`, `setup.py`, `setup.cfg`) with CI workflows for package tests, publishing, benchmarks, and integration tests.
   - `pdoc/` generates API docs, and `benchmarks/` measures parser and optimizer performance across representative SQL workloads.
   - The repo’s maintenance workflow is code-centric: tests, docs, and benchmark gates validate changes to the core AST/dialect pipeline.

14. **Architecture summary**
   - The essential boundary is the AST: everything funnels through it.
   - SQL text is normalized into a dialect-aware tree, manipulated by optimizer/analysis passes, then either rendered back to SQL or executed in-memory.
   - Dialects, optimizer rules, and the executor are the three main extension points around that central model.

Main entry points: [`sqlglot/expressions`](https://github.com/tobymao/sqlglot/tree/main/sqlglot/expressions) (AST), [`sqlglot/__init__.py`](https://github.com/tobymao/sqlglot/blob/main/sqlglot/__init__.py) (sqlglot), [`sqlglot/parser.py`](https://github.com/tobymao/sqlglot/blob/main/sqlglot/parser.py) (Parser), [`sqlglot/generator.py`](https://github.com/tobymao/sqlglot/blob/main/sqlglot/generator.py) (Generator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core library"]
  node_package["sqlglot<br/>library pkg<br/>[__init__.py]"]
  node_tokens["Tokens<br/>lex model<br/>[tokens.py]"]
  node_tokenizer["Tokenizer<br/>lexing<br/>[tokenizer_core.py]"]
  node_parser["Parser<br/>[parser.py]"]
  node_expressions["AST<br/>expression tree"]
  node_generator["Generator<br/>sql renderer<br/>[generator.py]"]
  node_optimizer["Optimizer<br/>rewrite engine"]
  node_schema["Schema<br/>metadata<br/>[schema.py]"]
  node_lineage["Lineage<br/>analysis<br/>[lineage.py]"]
  node_diff["Diff<br/>comparison<br/>[diff.py]"]
  node_typing["Typing<br/>type helpers"]
  node_helpers["Helpers<br/>support code<br/>[helper.py]"]
end

subgraph group_dialects["Dialect layer"]
  node_parsers["Dialect parsers<br/>grammar extensions"]
  node_dialects["Dialects<br/>dialect registry"]
  node_generators["Dialect generators<br/>output rules"]
end

subgraph group_runtime["Runtime paths"]
  node_executor["Executor<br/>in-memory engine"]
  node_sqlglotc["sqlglotc<br/>compiled accel"]
end

subgraph group_tooling["Tooling"]
  node_benchmarks["Benchmarks<br/>performance"]
  node_docs["Docs<br/>documentation"]
end

subgraph group_tests["Tests"]
  node_tests["Tests<br/>validation"]
end

node_package -->|"exports"| node_tokenizer
node_package -->|"exports"| node_parser
node_package -->|"exports"| node_generator
node_package -->|"exports"| node_optimizer
node_tokenizer -->|"uses"| node_tokens
node_tokenizer -->|"feeds"| node_parser
node_parser -->|"extends"| node_parsers
node_parsers -->|"builds"| node_expressions
node_dialects -->|"configures"| node_parsers
node_dialects -->|"configures"| node_generators
node_dialects -->|"configures"| node_typing
node_expressions -->|"renders"| node_generator
node_generators -->|"implements"| node_generator
node_expressions -->|"rewritten by"| node_optimizer
node_schema -->|"annotates"| node_optimizer
node_expressions -->|"inspected by"| node_lineage
node_expressions -->|"compared by"| node_diff
node_expressions -->|"interpreted by"| node_executor
node_helpers -->|"supports"| node_tokenizer
node_helpers -->|"supports"| node_generator
node_sqlglotc -.->|"accelerates"| node_package
node_benchmarks -.->|"measures"| node_parser
node_benchmarks -.->|"measures"| node_optimizer
node_docs -.->|"documents"| node_expressions
node_tests -.->|"verifies"| node_parser
node_tests -.->|"verifies"| node_generator
node_tests -.->|"verifies"| node_optimizer
node_tests -.->|"verifies"| node_executor

click node_package "https://github.com/tobymao/sqlglot/blob/main/sqlglot/__init__.py"
click node_tokens "https://github.com/tobymao/sqlglot/blob/main/sqlglot/tokens.py"
click node_tokenizer "https://github.com/tobymao/sqlglot/blob/main/sqlglot/tokenizer_core.py"
click node_parser "https://github.com/tobymao/sqlglot/blob/main/sqlglot/parser.py"
click node_parsers "https://github.com/tobymao/sqlglot/tree/main/sqlglot/parsers"
click node_expressions "https://github.com/tobymao/sqlglot/tree/main/sqlglot/expressions"
click node_dialects "https://github.com/tobymao/sqlglot/tree/main/sqlglot/dialects"
click node_generator "https://github.com/tobymao/sqlglot/blob/main/sqlglot/generator.py"
click node_generators "https://github.com/tobymao/sqlglot/tree/main/sqlglot/generators"
click node_optimizer "https://github.com/tobymao/sqlglot/tree/main/sqlglot/optimizer"
click node_schema "https://github.com/tobymao/sqlglot/blob/main/sqlglot/schema.py"
click node_lineage "https://github.com/tobymao/sqlglot/blob/main/sqlglot/lineage.py"
click node_diff "https://github.com/tobymao/sqlglot/blob/main/sqlglot/diff.py"
click node_executor "https://github.com/tobymao/sqlglot/tree/main/sqlglot/executor"
click node_typing "https://github.com/tobymao/sqlglot/tree/main/sqlglot/typing"
click node_helpers "https://github.com/tobymao/sqlglot/blob/main/sqlglot/helper.py"
click node_sqlglotc "https://github.com/tobymao/sqlglot/tree/main/sqlglotc"
click node_benchmarks "https://github.com/tobymao/sqlglot/tree/main/benchmarks"
click node_docs "https://github.com/tobymao/sqlglot/tree/main/pdoc"
click node_tests "https://github.com/tobymao/sqlglot/tree/main/tests"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_package,node_tokens,node_tokenizer,node_parser,node_expressions,node_generator,node_optimizer,node_schema,node_lineage,node_diff,node_typing,node_helpers toneBlue
class node_parsers,node_dialects,node_generators toneAmber
class node_executor,node_sqlglotc toneMint
class node_benchmarks,node_docs toneRose
class node_tests toneIndigo
```

## Components

### Core library

Primary SQL pipeline centered on AST creation, rewrites, rendering, and analysis.

- [sqlglot](https://github.com/tobymao/sqlglot/blob/main/sqlglot/__init__.py): library pkg · Public package surface that ties together parsing, transpiling, optimization, and analysis APIs. · `sqlglot/__init__.py`
- [Tokens](https://github.com/tobymao/sqlglot/blob/main/sqlglot/tokens.py): lex model · Token definitions, keyword lookup, and lexical helpers. · `sqlglot/tokens.py`
- [Tokenizer](https://github.com/tobymao/sqlglot/blob/main/sqlglot/tokenizer_core.py): lexing · Converts SQL text into tokens. · `sqlglot/tokenizer_core.py`
- [Parser](https://github.com/tobymao/sqlglot/blob/main/sqlglot/parser.py): syntax parser · Shared parser entry point that builds the internal SQL tree. · `sqlglot/parser.py`
- [AST](https://github.com/tobymao/sqlglot/tree/main/sqlglot/expressions): expression tree · Central SQL expression model used by parsing, rewriting, rendering, and execution. · `sqlglot/expressions`
- [Generator](https://github.com/tobymao/sqlglot/blob/main/sqlglot/generator.py): sql renderer · Renders AST nodes back into SQL text. · `sqlglot/generator.py`
- [Optimizer](https://github.com/tobymao/sqlglot/tree/main/sqlglot/optimizer): rewrite engine · Rule-based AST rewrites for qualification, normalization, pushdown, and simplification. · `sqlglot/optimizer`
- [Schema](https://github.com/tobymao/sqlglot/blob/main/sqlglot/schema.py): metadata · Table and column metadata used by analysis and optimization. · `sqlglot/schema.py`
- [Lineage](https://github.com/tobymao/sqlglot/blob/main/sqlglot/lineage.py): analysis · Traces output columns back to source expressions and tables. · `sqlglot/lineage.py`
- [Diff](https://github.com/tobymao/sqlglot/blob/main/sqlglot/diff.py): comparison · Computes semantic differences between SQL trees. · `sqlglot/diff.py`
- [Typing](https://github.com/tobymao/sqlglot/tree/main/sqlglot/typing): type helpers · Dialect-specific type logic and annotations. · `sqlglot/typing`
- [Helpers](https://github.com/tobymao/sqlglot/blob/main/sqlglot/helper.py): support code · Shared support for errors, serialization, JSONPath, time, tries, and general utilities. · `sqlglot/helper.py`

### Dialect layer

Dialect-specific parsing, generation, and typing behavior.

- [Dialect parsers](https://github.com/tobymao/sqlglot/tree/main/sqlglot/parsers): grammar extensions · Per-dialect parsing rules and syntax specialization. · `sqlglot/parsers`
- [Dialects](https://github.com/tobymao/sqlglot/tree/main/sqlglot/dialects): dialect registry · Dialect classes that bundle tokenizer, parser, generator, and type behavior. · `sqlglot/dialects`
- [Dialect generators](https://github.com/tobymao/sqlglot/tree/main/sqlglot/generators): output rules · Per-dialect SQL rendering and syntax mapping. · `sqlglot/generators`

### Runtime paths

In-memory execution and optional compiled acceleration.

- [Executor](https://github.com/tobymao/sqlglot/tree/main/sqlglot/executor): in-memory engine · Python SQL execution over in-memory tables using the AST. · `sqlglot/executor`
- [sqlglotc](https://github.com/tobymao/sqlglot/tree/main/sqlglotc): compiled accel · Optional mypyc-backed acceleration for the same library surface. · `sqlglotc`

### Tooling

Docs, benchmarks, packaging, and release support.

- [Benchmarks](https://github.com/tobymao/sqlglot/tree/main/benchmarks): performance · Parser and optimizer performance measurement harness. · `benchmarks`
- [Docs](https://github.com/tobymao/sqlglot/tree/main/pdoc): documentation · API docs and explanatory posts for the library architecture. · `pdoc`

### Tests

Subsystem-focused validation and fixture coverage.

- [Tests](https://github.com/tobymao/sqlglot/tree/main/tests): validation · Subsystem-focused tests and integration coverage. · `tests`

## Connections

- sqlglot → Tokenizer: exports · Public API exposes the tokenizer layer.
- sqlglot → Parser: exports · Public API exposes parsing entry points.
- sqlglot → Generator: exports · Public API exposes SQL rendering and transpilation.
- sqlglot → Optimizer: exports · Public API exposes rewrite and canonicalization routines.
- Tokenizer → Tokens: uses · Tokenizer relies on token and keyword definitions.
- Tokenizer → Parser: feeds · Token stream is consumed by the parser.
- Parser → Dialect parsers: extends · Shared parser delegates dialect-specific grammar rules.
- Dialect parsers → AST: builds · Dialect parsing produces the shared AST.
- Dialects → Dialect parsers: configures · Dialect classes select parser behavior and syntax.
- Dialects → Dialect generators: configures · Dialect classes select generator behavior and output rules.
- Dialects → Typing: configures · Dialect classes also control type-specific semantics.
- AST → Generator: renders · AST nodes are rendered back to SQL text.
- Dialect generators → Generator: implements · Per-dialect renderers specialize the shared generator.
- AST → Optimizer: rewritten by · Optimizer transforms the AST into canonical forms.
- Schema → Optimizer: annotates · Metadata informs qualification and type-sensitive rewrites.
- AST → Lineage: inspected by · Lineage walks the AST to trace dependencies.
- AST → Diff: compared by · Diff compares semantic structure between trees.
- AST → Executor: interpreted by · Executor evaluates the parsed tree in memory.
- Helpers → Tokenizer: supports · Helpers support parsing, errors, and lookup utilities.
- Helpers → Generator: supports · Helpers support serialization and formatting concerns.
- sqlglotc → sqlglot: accelerates · Compiled package accelerates the same public library surface.
- Benchmarks → Parser: measures · Benchmarks cover parsing workloads.
- Benchmarks → Optimizer: measures · Benchmarks cover optimization workloads.
- Docs → AST: documents · Documentation explains the AST and API surface.
- Tests → Parser: verifies · Parser tests validate syntax and dialect behavior.
- Tests → Generator: verifies · Generator tests validate SQL output.
- Tests → Optimizer: verifies · Optimizer tests validate rewrite rules.
- Tests → Executor: verifies · Executor tests validate in-memory evaluation.
