# geldata/gel architecture

> Architecture diagram of the GitHub repository geldata/gel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/geldata/gel
- Repository: https://github.com/geldata/gel

## Overview

##### Main Components
- Client/Interface Layer:
- Command line tool (see edb/cli, build_backend.py, setup.py)
- Client libraries (e.g. in docs/intro/quickstart and edb/clients in the documentation)
- Web-based interfaces if any (there’s a reference to GraphQL and HTTP endpoints in edb/graphql and rust/gel-http)

- Query Parsing and Compilation:
- EdgeQL Parser (edb/edgeql-parser and its Cargo.toml based “edgeql-parser-python” subproject)
- Query compiler components (edb/edgeql/compiler, edb/pgsql/compiler, edb/server/compiler modules)
- Modules for the new query language EdgeQL that transform user queries into database execution plans

- Server/Execution Engine:
- The core server (edb/server), which handles connections, processes EdgeQL queries, enforces authentication (see edb/server/auth, edb/server/connpool modules), and dispatches compiled queries.
- The protocol layer (edb/protocol and edb/server/protocol) which manages client/server communication, including authentication (edb/server/protocol/auth_ext) and query execution.
- The connection pool and multi-tenant/runtime infrastructure (see edb/server/connpool, rust/conn_pool)

- Storage & Database Back-End Integration:
- Components in edb/pgsql (which deal with translating compiled queries to PostgreSQL-compatible SQL, database ops, and related schema migration/DML code)
- Schema, datamodel, and migration support (edb/schema and parts of the docs referencing ESDL files)

- Ancillary & Supporting Tools:
- Test suites and integration tests wrapped in the tests directory.
- DevOps and deployment scripts — e.g., GitHub workflows, Terraform scripts in .github/aws-aurora, .github/aws-rds, etc.
- Rust-based projects (rust/db_proto, rust/gel-http, rust/pgrust, rust/pyo3_util) that offer low-level protocol implementations and connection logic.
- Experimental interpreter and additional tooling (edb/tools, edb/tools/experimental_interpreter)

##### Relationships and Data Flows
- Route these queries into the Client/Interface Layer, which then passes them on via the network to the Server Layer.
- The Server processes the requests by first handing them off to the Query Parser component (EdgeQL Parser) to validate and tokenize the syntax.
- Next, the Query Compiler builds an execution plan. This may involve a transformation through the compiler modules – note the separation between the “edgeql” and “pgsql” compilers.
- The execution plan is then processed by the Server/Execution Engine using a connection pool to interact with the underlying storage/backend (typically PostgreSQL as suggested by edb/pgsql).
- The results flow back through the protocol layer to the Client/Interface Layer where they are returned to the originating client.
- Ancillary modules provide metrics, logging, error handling, migration support, and schema introspection.
- The supporting Rust-based modules should be depicted as a “fast-path” or “lower-level” optimization layer for critical parts like connection pooling and protocol message handling.

##### Architectural Patterns and Technologies
- Separation of concerns between the query interface (EdgeQL), its parsing/compilation engine, and the backend execution. This separation maps closely to a layered architecture similar to MVC (but for a database engine).
- Clear division between modules written in Python and Rust (for performance-critical backend functionalities).
- Use of workflows/CI (GitHub workflows and Terraform for infrastructure deployment) and accompanying documentation (docs, dev-notes) that support continuous integration/deployment.

##### Final Elements to Incorporate
- Clear and concise labeling of each major component.

## Diagram (Mermaid)

```mermaid
flowchart TD
    U["User Query"]:::external

    subgraph "Client/Interface Layer"
        CLI["Command Line Tool"]:::client
        Build["Build Backend Script"]:::client
        Setup["Setup Script"]:::client
        CLib["Client Libraries & Quickstart"]:::client
        GQL["GraphQL Interface"]:::client
        HTTP["HTTP Interface"]:::client
    end

    subgraph "Query Parsing & Compilation"
        Parser["EdgeQL Parser"]:::compiler
        EdgeQLComp["EdgeQL Compiler"]:::compiler
        PSQLComp["PostgreSQL Compiler"]:::compiler
        ServerComp["Server Compiler"]:::compiler
    end

    subgraph "Server/Execution Engine"
        Core["Core Server"]:::server
        Auth["Authentication & Connection"]:::server
        ConnPy["Connection Pool (Python)"]:::server
        ConnRust["Connection Pool (Rust)"]:::server
        Proto["Protocol Layer"]:::server
        ProtoAuth["Protocol Auth"]:::server
        ProtoAuthExt["Protocol Auth Ext"]:::server
    end

    subgraph "Storage & Back-End Integration"
        PSQLAdapter["PostgreSQL Adapter"]:::storage
        Schema["Schema & Migration"]:::storage
    end

    subgraph "Ancillary & Supporting Tools"
        Tests["Test Suites"]:::ancillary
        DevOps["DevOps & Deployment"]:::ancillary
        RustMods["Rust Modules"]:::ancillary
        ExpInterp["Experimental Interpreter"]:::ancillary
    end

    U --> CLI
    U --> CLib
    U --> GQL
    U --> HTTP

    CLI --> Parser
    CLib --> Parser
    GQL --> Parser
    HTTP --> Parser

    Parser --> EdgeQLComp
    EdgeQLComp --> PSQLComp
    PSQLComp --> ServerComp

    ServerComp --> Core
    Core --> Auth
    Auth --> ConnPy
    ConnPy --> Proto
    Proto --> ConnRust
    ConnRust --> PSQLAdapter
    PSQLAdapter --> Schema

    Schema --> Core
    Core --> U

    Proto --> ProtoAuth
    Proto --> ProtoAuthExt

    Tests --- Core
    DevOps --- Core
    RustMods --- ConnRust
    ExpInterp --- ServerComp

    click CLI "https://github.com/geldata/gel/tree/master/edb/cli"
    click Build "https://github.com/geldata/gel/blob/master/build_backend.py"
    click Setup "https://github.com/geldata/gel/blob/master/setup.py"
    click CLib "https://github.com/geldata/gel/tree/master/docs/intro/quickstart"
    click GQL "https://github.com/geldata/gel/tree/master/edb/graphql"
    click HTTP "https://github.com/geldata/gel/tree/master/rust/gel-http"

    click Parser "https://github.com/geldata/gel/tree/master/edb/edgeql-parser"
    click EdgeQLComp "https://github.com/geldata/gel/tree/master/edb/edgeql/compiler"
    click PSQLComp "https://github.com/geldata/gel/tree/master/edb/pgsql/compiler"
    click ServerComp "https://github.com/geldata/gel/tree/master/edb/server/compiler"

    click Core "https://github.com/geldata/gel/tree/master/edb/server"
    click Auth "https://github.com/geldata/gel/tree/master/edb/server/auth"
    click ConnPy "https://github.com/geldata/gel/tree/master/edb/server/connpool"
    click Proto "https://github.com/geldata/gel/tree/master/edb/protocol"
    click ProtoAuth "https://github.com/geldata/gel/tree/master/edb/server/protocol/auth"
    click ProtoAuthExt "https://github.com/geldata/gel/tree/master/edb/server/protocol/auth_ext"
    click ConnRust "https://github.com/geldata/gel/tree/master/rust/conn_pool"

    click PSQLAdapter "https://github.com/geldata/gel/tree/master/edb/pgsql"
    click Schema "https://github.com/geldata/gel/tree/master/edb/schema"

    click Tests "https://github.com/geldata/gel/tree/master/tests"
    click DevOps "https://github.com/geldata/gel/blob/master/.github"
    click RustMods "https://github.com/geldata/gel/tree/master/rust/db_proto"
    click ExpInterp "https://github.com/geldata/gel/tree/master/edb/tools/experimental_interpreter"

    classDef external fill:#ffd,stroke:#333,stroke-width:2px;
    classDef client fill:#ffcccc,stroke:#600,stroke-width:2px;
    classDef compiler fill:#ccf2ff,stroke:#006,stroke-width:2px;
    classDef server fill:#ccffcc,stroke:#060,stroke-width:2px;
    classDef storage fill:#fff0b3,stroke:#996600,stroke-width:2px;
    classDef ancillary fill:#e6e6e6,stroke:#666,stroke-width:2px;
```
