# kensho-technologies/graphql-compiler architecture

> Architecture diagram of the GitHub repository kensho-technologies/graphql-compiler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kensho-technologies/graphql-compiler
- Repository: https://github.com/kensho-technologies/graphql-compiler

## Overview

##### Project type and purpose
- This is not a full-stack app; it’s an open-source compiler library (“graphql-compiler”) whose sole purpose is to translate GraphQL queries into optimized database queries (SQL, Cypher/MATCH, Gremlin).

##### High-level architecture

A. Client / API Layer
- Entry points:
– `graphql_to_sql(...)`
– `graphql_to_match(...)`
– `graphql_to_gremlin(...)`
- Helpers to build schema info:
– SQL: `get_sqlalchemy_schema_info(...)`
– OrientDB: `get_graphql_schema_from_orientdb_schema_data(...)`

B. Compiler Frontend
- Parses GraphQL query (using `graphql` lib).
- Performs initial AST manipulation & validation (`compiler_frontend.py`, `validation.py`).
- Outputs an intermediate representation (IR).

C. Macro Expansion & Transformations
- Expand user-defined macros (`macros/`)
- Schema transformations (rename, split, merge) (`schema_transformation/`).

D. IR Optimization & Planning
- Apply consistency checks, common IR optimizations (`ir_lowering_common/`).
- Cost Estimation modules (`cost_estimation/`) compute selectivities, cardinalities.
- Pagination planning (`query_pagination/`).

E. IR Lowering per Backend
For each target language there is:
- A lowering pipeline (`ir_lowering_sql/`, `ir_lowering_cypher/`, `ir_lowering_gremlin/`, `ir_lowering_match/`).
- Emitters (`emit_sql.py`, `emit_cypher.py`, `emit_gremlin.py`, `emit_match.py`) that walk lowered IR and produce query strings.

F. Post-processing
- Target-specific tweaks (e.g. SQL post_processing).

G. Execution (optional)
- SQLAlchemy engine (`create_engine`).
- OrientDB / RedisGraph / Neo4j drivers via `graphql_compiler.api.*`.

H. Testing & Tooling
- End-to-end & unit tests (`tests/`).
- CLI entry point: `graphql_compiler/tool.py`.
- Docker Compose to spin up test databases.

##### Relationships and data flow
Client/API → Frontend → Macro Expansion → IR Optimizer/Planner → IR Lowering → Emit → Post-processing → Generated Query → Database Execution

##### Architectural patterns and principles
- Compiler pipeline / visitor pattern for AST and IR.
- Pluggable back-end modules.
- Single responsibility: each Python subpackage (e.g., `query_formatting`, `schema_generation`) has a focused concern.
- Separation of concerns: front-end vs. lowering vs. emitter vs. post-processing.

##### Final checklist
– All major compiler subpackages (`compiler`, `macros`, `schema_generation`, `query_planning`, `ir_lowering_*`, `emit_*`, `post_processing`).
– API layer (`graphql_to_*` functions, `get_*_schema_info`).
– External DB back-ends.
– Clear labels and callouts for key modules (e.g., “Cost Estimation: determines filter selectivities”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client/API Layer
    A1(["API Layer\ngraphql_to_sql(),\ngraphql_to_match(),\ngraphql_to_gremlin()"]):::api
    click A1 "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/backend.py"

    %% Schema Helpers
    subgraph "Schema Generation" 
        direction TB
        SH1["SQLAlchemy Schema Introspection"]:::schema
        click SH1 "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/schema_generation/sqlalchemy/schema_graph_builder.py"
        SH2["OrientDB Schema Introspection"]:::schema
        click SH2 "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/schema_generation/orientdb/schema_graph_builder.py"
    end

    %% Compiler Frontend
    subgraph "Compiler Frontend"
        direction TB
        FE1["Parse & Initial AST (compiler_frontend.py)"]:::phase
        click FE1 "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/compiler/compiler_frontend.py"
        FE2["Validation Utilities (validation.py)"]:::phase
        click FE2 "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/compiler/validation.py"
    end

    %% Macro Expansion & Transformations
    subgraph "Macro Expansion & Schema Transformations"
        direction TB
        ME["Macro Expansion (macro_expansion.py)"]:::phase
        click ME "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/macros/macro_expansion.py"
        ME2["User-Defined Macros (macros/macro_edge/)"]:::phase
        click ME2 "https://github.com/kensho-technologies/graphql-compiler/tree/main/graphql_compiler/macros/macro_edge/"
        ST["Schema Transformations\n(rename, split, merge)"]:::phase
        click ST "https://github.com/kensho-technologies/graphql-compiler/tree/main/graphql_compiler/schema_transformation/"
    end

    %% IR Optimization & Planning
    subgraph "IR Optimization & Planning"
        direction TB
        IR1["Common Lowering Helpers"]:::phase
        click IR1 "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/compiler/ir_lowering_common/common.py"
        IR2["Cost Estimation Modules"]:::phase
        click IR2 "https://github.com/kensho-technologies/graphql-compiler/tree/main/graphql_compiler/cost_estimation/"
        IR3["Pagination Planning"]:::phase
        click IR3 "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/query_pagination/pagination_planning.py"
        IR4["Query Planning"]:::phase
        click IR4 "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/query_planning/make_query_plan.py"
    end

    %% Backend Selection
    SB{"Select Backend"}:::decision

    %% IR Lowering per Backend
    subgraph "IR Lowering Pipelines" 
        direction TB
        LSQL["SQL Lowering"]:::backend
        click LSQL "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/compiler/ir_lowering_sql/ir_lowering.py"
        LCYP["Cypher Lowering"]:::backend
        click LCYP "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/compiler/ir_lowering_cypher/ir_lowering.py"
        LGRE["Gremlin Lowering"]:::backend
        click LGRE "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/compiler/ir_lowering_gremlin/ir_lowering.py"
        LMAT["MATCH Lowering"]:::backend
        click LMAT "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/compiler/ir_lowering_match/ir_lowering.py"
    end

    %% Emitters
    subgraph "Emitters"
        direction TB
        ESQL["SQL Emitter"]:::backend
        click ESQL "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/compiler/emit_sql.py"
        ECYP["Cypher Emitter"]:::backend
        click ECYP "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/compiler/emit_cypher.py"
        EGRE["Gremlin Emitter"]:::backend
        click EGRE "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/compiler/emit_gremlin.py"
        EMAT["MATCH Emitter"]:::backend
        click EMAT "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/compiler/emit_match.py"
    end

    %% Post-Processing
    PP["SQL Post-Processing Tweaks"]:::phase
    click PP "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/post_processing/sql_post_processing.py"

    %% Generated Query
    GQ(["Generated Query String"]):::phase

    %% Database Execution
    subgraph "Execution & Drivers"
        direction TB
        DB1["Relational DB Engines"]:::external
        click DB1 "https://github.com/kensho-technologies/graphql-compiler/tree/main/graphql_compiler/api/sql/"
        DB2["OrientDB Driver"]:::external
        click DB2 "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/api/orientdb.py"
        DB3["RedisGraph Driver"]:::external
        click DB3 "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/api/redisgraph.py"
    end

    %% Testing & Tooling
    subgraph "Tooling & Testing"
        direction TB
        CLI["CLI Entry Point"]:::tool
        click CLI "https://github.com/kensho-technologies/graphql-compiler/blob/main/graphql_compiler/tool.py"
        TEST["Tests & CI"]:::tool
        click TEST "https://github.com/kensho-technologies/graphql-compiler/tree/main/tests/"
        click TEST "https://github.com/kensho-technologies/graphql-compiler/blob/main/docker-compose.yml"
    end

    %% Connections
    A1 --> SH1
    A1 --> SH2
    SH1 --> FE1
    SH2 --> FE1
    FE1 --> FE2
    FE2 --> ME
    ME --> ME2
    ME2 --> ST
    ST --> IR1
    IR1 --> IR2
    IR2 --> IR3
    IR3 --> IR4
    IR4 --> SB
    SB --> LSQL
    SB --> LCYP
    SB --> LGRE
    SB --> LMAT
    LSQL --> ESQL
    LCYP --> ECYP
    LGRE --> EGRE
    LMAT --> EMAT
    ESQL --> PP
    ECYP --> GQ
    EGRE --> GQ
    EMAT --> GQ
    PP --> GQ
    GQ --> DB1
    GQ --> DB2
    GQ --> DB3
    CLI --> A1
    TEST --> A1

    %% Styles
    classDef api fill:#d0ebff,stroke:#74c0fc,stroke-width:2px;
    classDef schema fill:#fff3bf,stroke:#fcc419,stroke-width:2px;
    classDef phase fill:#e7f5ff,stroke:#339af0,stroke-width:2px;
    classDef backend fill:#e6fcf5,stroke:#38d9a9,stroke-width:2px;
    classDef decision fill:#ffe3e3,stroke:#fa5252,stroke-width:2px,stroke-dasharray: 5 5;
    classDef external fill:#f1f3f5,stroke:#868e96,stroke-width:2px,stroke-dasharray: 2 2;
    classDef tool fill:#dee2ff,stroke:#5c7cfa,stroke-width:2px;
```
