# apache/datafusion-sqlparser-rs architecture

> Architecture diagram of the GitHub repository apache/datafusion-sqlparser-rs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/datafusion-sqlparser-rs
- Repository: https://github.com/apache/datafusion-sqlparser-rs

## Overview

##### Main Components to Highlight
- Tokenizer: Located in src/tokenizer.rs, which breaks SQL input into tokens.
- Parser: Major parsing logic found in src/parser and src/ast, with AST node definitions (src/ast/*) and helper modules. Indicate that the parser implements both a recursive descent approach for statements and a Pratt parser for expression handling.
- SQL Dialects: Create a component for the dialect modules (located in src/dialect). Within this, details for specific dialects (ANSI, BigQuery, ClickHouse, etc.) should be shown, signaling the project’s extensibility for various SQL standards.
- AST (Abstract Syntax Tree) Module: Include a distinct AST component that aggregates nodes from src/ast. Emphasize that AST nodes (and related utilities like visitors if enabled by the 'visitor' feature) are a central part of the design.
- CLI Application & Examples: Mark a component for the CLI tool (as seen in the examples and the command line section of the README) that demonstrates how to use the parser to output an AST (even in JSON format if the `json_example` feature is enabled).
- Developer Tooling: Include a module/section for developer and release tooling (found inside dev/release and related scripts) plus CI/CD configuration (e.g. GitHub Actions in the .github folder).

##### Relationships and Interactions
- Data flow starting from SQL input going to the Tokenizer, then to the Parser, and finally to an AST representation.
- Integration points where optional features (like Serde serialization and AST visitors) interact with the AST generation or processing.

##### Architectural Patterns and Design Principles
- That the parser uses the Pratt Parser design (highlight an annotation that references the “Top-Down Operator Precedence” pattern) for processing expressions.
- A modular and layered approach where the core lexer and parser are independent of individual SQL dialect implementations.
- That the project is designed for extensibility (allowing custom SQL dialects) and integration (e.g., via CLI or as an embedded library in larger query engines).

##### Technologies, Frameworks, and Libraries
- Rust is the programming language – show the Cargo build system controlling compilation and dependency management (Cargo.toml at the root and in sub-crates like derive).
- Optional features include Serde (for AST serialization) and the use of third-party libraries such as the recursive protection crate.
- That GitHub CI/CD workflows are used (configurations in the .github folders).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Parsing Engine
    A["SQL Input"]:::core
    B["Tokenizer"]:::core
    C["Parser (Recursive Descent & Pratt)"]:::core
    D["Entry Point (lib.rs)"]:::core
    E["AST Module"]:::core
    F["SQL Dialects"]:::dialect

    %% Auxiliary Components
    G["CLI Tool"]:::cli
    H["Tests"]:::testing
    I["Benchmarks"]:::testing
    J["Developer Tooling/Release Scripts"]:::dev
    K["CI/CD (.github)"]:::dev

    %% Connections within Core Parsing Engine
    A -->|"tokenizes"| B
    B -->|"parses"| C
    D -->|"initializes"| C
    C -->|"buildsAST"| E
    C <-->|"consults"| F

    %% Integration with Auxiliary Components
    G -->|"invokes"| C
    H -->|"validates"| C
    I -->|"benchmarks"| C
    J -->|"triggers"| K
    K -->|"runsTests"| H

    %% Click Events
    click B "https://github.com/apache/datafusion-sqlparser-rs/blob/main/src/tokenizer.rs"
    click C "https://github.com/apache/datafusion-sqlparser-rs/tree/main/src/parser"
    click D "https://github.com/apache/datafusion-sqlparser-rs/blob/main/src/lib.rs"
    click E "https://github.com/apache/datafusion-sqlparser-rs/tree/main/src/ast"
    click F "https://github.com/apache/datafusion-sqlparser-rs/tree/main/src/dialect"
    click G "https://github.com/apache/datafusion-sqlparser-rs/blob/main/examples/cli.rs"
    click H "https://github.com/apache/datafusion-sqlparser-rs/tree/main/tests"
    click I "https://github.com/apache/datafusion-sqlparser-rs/tree/main/sqlparser_bench"
    click J "https://github.com/apache/datafusion-sqlparser-rs/tree/main/dev/release"
    click K "https://github.com/apache/datafusion-sqlparser-rs/blob/main/.github"

    %% Styles
    classDef core fill:#E1F5FE,stroke:#0277BD,stroke-width:2px;
    classDef dialect fill:#F1F8E9,stroke:#33691E,stroke-width:2px;
    classDef cli fill:#FFF3E0,stroke:#EF6C00,stroke-width:2px;
    classDef testing fill:#FCE4EC,stroke:#C2185B,stroke-width:2px;
    classDef dev fill:#EDE7F6,stroke:#5E35B1,stroke-width:2px;
```
