How geldata/gel works
Main Components
• Client/Interface Layer:
• Command line tool (see edb/cli, build_backend.py, setup.py)
• Client libraries (e.g. in docs/intro/quickstart and edb/clients in the documentation)
• Web-based interfaces if any (there’s a reference to GraphQL and HTTP endpoints in edb/graphql and rust/gel-http)
• Query Parsing and Compilation:
• EdgeQL Parser (edb/edgeql-parser and its Cargo.toml based “edgeql-parser-python” subproject)
• Query compiler components (edb/edgeql/compiler, edb/pgsql/compiler, edb/server/compiler modules)
• Modules for the new query language EdgeQL that transform user queries into database execution plans
• Server/Execution Engine:
• The core server (edb/server), which handles connections, processes EdgeQL queries, enforces authentication (see edb/server/auth, edb/server/connpool modules), and dispatches compiled queries.
• The protocol layer (edb/protocol and edb/server/protocol) which manages client/server communication, including authentication (edb/server/protocol/auth_ext) and query execution.
• The connection pool and multi-tenant/runtime infrastructure (see edb/server/connpool, rust/conn_pool)
• Storage & Database Back-End Integration:
• Components in edb/pgsql (which deal with translating compiled queries to PostgreSQL-compatible SQL, database ops, and related schema migration/DML code)
• Schema, datamodel, and migration support (edb/schema and parts of the docs referencing ESDL files)
• Ancillary & Supporting Tools:
• Test suites and integration tests wrapped in the tests directory.
• DevOps and deployment scripts — e.g., GitHub workflows, Terraform scripts in .github/aws-aurora, .github/aws-rds, etc.
• Rust-based projects (rust/db_proto, rust/gel-http, rust/pgrust, rust/pyo3_util) that offer low-level protocol implementations and connection logic.
• Experimental interpreter and additional tooling (edb/tools, edb/tools/experimental_interpreter)
Relationships and Data Flows
• Route these queries into the Client/Interface Layer, which then passes them on via the network to the Server Layer.
• The Server processes the requests by first handing them off to the Query Parser component (EdgeQL Parser) to validate and tokenize the syntax.
• Next, the Query Compiler builds an execution plan. This may involve a transformation through the compiler modules – note the separation between the “edgeql” and “pgsql” compilers.
• The execution plan is then processed by the Server/Execution Engine using a connection pool to interact with the underlying storage/backend (typically PostgreSQL as suggested by edb/pgsql).
• The results flow back through the protocol layer to the Client/Interface Layer where they are returned to the originating client.
• Ancillary modules provide metrics, logging, error handling, migration support, and schema introspection.
• The supporting Rust-based modules should be depicted as a “fast-path” or “lower-level” optimization layer for critical parts like connection pooling and protocol message handling.
Architectural Patterns and Technologies
• Separation of concerns between the query interface (EdgeQL), its parsing/compilation engine, and the backend execution. This separation maps closely to a layered architecture similar to MVC (but for a database engine).
• Clear division between modules written in Python and Rust (for performance-critical backend functionalities).
• Use of workflows/CI (GitHub workflows and Terraform for infrastructure deployment) and accompanying documentation (docs, dev-notes) that support continuous integration/deployment.
Final Elements to Incorporate
• Clear and concise labeling of each major component.