# xorq-labs/xorq architecture

> Architecture diagram of the GitHub repository xorq-labs/xorq, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xorq-labs/xorq
- Repository: https://github.com/xorq-labs/xorq

## Overview

##### Main Architectural Components
- • CLI & User Interface: The command-line interface (as seen in python/xorq/cli.py and the documented “xorq build” commands) that allows users to build expression artifacts.
- • Python Library Layer (Frontend for Pipeline Definitions): This layer is responsible for receiving user-created transformations (using deferred expressions, UDFs, etc.). It organizes transformations and converts them into Ibis expressions.
- • Deferred Execution & Serialization Engine: This component handles the creation and serialization of pipelines (see the build artifacts and YAML/SQL outputs) ensuring pipelines are reproducible with built-in caching.
- • Backends (Multi-engine Integration): There are multiple backend components under python/xorq/backends for engines like DataFusion, DuckDB, Postgres, Snowflake, and Trino. Each backend includes modules for compiling, executing, or even caching pipelines.
- • Caching System: Embedded caching functions (see python/xorq/caching and related tests) that cache and reuse intermediate results.
- • UDF & UDXF System: Modules for handling user-defined functions and user-defined aggregates (both in Python and possibly integrated via the Rust core) to enable computations that are portable across engines.
- • Rust Core & Execution Engine: Located in src/, containing the core implementation (query parsing, compiling, optimization, and execution) written in Rust.
- • External Services & Containerization: Docker configuration files and compose.yaml signal integration with external data engines/services (such as Postgres, MinIO for object storage, and Trino). This represents an environment where the framework can be deployed with supporting services.
- • Documentation & Examples: Although not part of the execution flow, the docs, examples, and tests offer insights into typical data flows and component interactions.

##### Relationships and Interactions
- The CLI and Python library components serve as the entry points for user pipelines.
- Once a pipeline is defined using declarative (Ibis-inspired) expressions, it is serialized (e.g., into YAML and SQL files) and then submitted to the underlying execution engine.
- The multi-engine backends receive these queries or expressions and translate them into engine-specific commands or execution plans.
- The caching system intercepts and stores intermediate results to optimize the computations.
- The Rust core coordinates with the Python layer, performing compile-time optimizations and executing the deferred expressions.
- External services (post deployment environments, typically containerized) interact with the backends to supply necessary data and processing capabilities.

##### Architectural Patterns and Design Principles
- Declarative Programming: Users write transformations in a high-level, pandas-like style, and the system manages deferred execution.
- Separation of Concerns: Clear separation between the pipeline definition (Python front-end), execution and caching (backend modules and Rust core), and deployment (Docker/external services).
- Multi-engine Integration: A pattern that abstracts away engine-specific details, allowing the same logic to execute on multiple data processing platforms.
- Serialization for Reproducibility: Pipelines are converted to SQL/YAML artifacts ensuring that the execution graph can be version controlled and consistently reproduced.

##### Relevant Technologies, Frameworks, and Libraries
- Python (for the user-facing API and CLI)
- Ibis (for abstract query expressions)
- DataFusion (as one of the execution engines)
- Rust (for high-performance query compilation and execution in the src/ directory)
- Docker (deployment and integration testing, as seen in docker/ and compose.yaml)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface Layer
    subgraph "User Interface Layer"
        CLI["CLI & User Interface"]:::python
    end

    %% Pipeline Definition & Serialization Layer
    subgraph "Pipeline Definition & Serialization"
        API["Python Library Layer: Pipeline API"]:::python
        Expr["Deferred Expressions"]:::python
        UDF["UDF & UDXF System"]:::python
        YAML["YAML Compiler"]:::python
        SQL["SQL Serializer"]:::python
    end

    %% Execution & Multi-engine Integration Layer
    subgraph "Execution & Integration"
        DF["DataFusion Backend"]:::python
        DUK["DuckDB Backend"]:::python
        PG["Postgres Backend"]:::python
        SF["Snowflake Backend"]:::python
        TRI["Trino Backend"]:::python
        CACH["Caching System"]:::python
        RUST["Rust Core & Execution Engine"]:::rust
    end

    %% Deployment & External Services Layer
    subgraph "Deployment & External Services"
        EXT["External Services & Containerization"]:::external
    end

    %% Documentation & Examples (Supplementary)
    subgraph "Documentation & Examples"
        DOC["Documentation & Examples"]:::external
    end

    %% Connections: User Interface to Pipeline Definition
    CLI -->|"triggers"| API

    %% Pipeline Definition internal flow
    API -->|"organizes"| Expr
    Expr -->|"uses"| UDF
    Expr -->|"generates"| YAML
    Expr -->|"generates"| SQL
    Expr -->|"intercepts"| CACH

    %% Serialization outputs to Backends
    YAML -->|"sends"| DF
    YAML -->|"sends"| DUK
    YAML -->|"sends"| PG
    YAML -->|"sends"| SF
    YAML -->|"sends"| TRI
    SQL  -->|"sends"| DF
    SQL  -->|"sends"| DUK
    SQL  -->|"sends"| PG
    SQL  -->|"sends"| SF
    SQL  -->|"sends"| TRI

    %% Backends interact with Rust Core for compilation/optimization
    DF  -->|"optimizes"| RUST
    DUK -->|"optimizes"| RUST
    PG  -->|"optimizes"| RUST
    SF  -->|"optimizes"| RUST
    TRI -->|"optimizes"| RUST

    %% UDF integration with Rust Core
    UDF -->|"integrates"| RUST

    %% External Services interacting with Backends
    EXT -->|"supports"| PG
    EXT -->|"supports"| TRI
    EXT -->|"supports"| DF
    EXT -->|"supports"| DUK
    EXT -->|"supports"| SF

    %% Styling Classes
    classDef python fill:#E0F7FA,stroke:#00796B,stroke-width:2px;
    classDef rust fill:#DCEDC8,stroke:#558B2F,stroke-width:2px;
    classDef external fill:#FFF9C4,stroke:#FBC02D,stroke-width:2px;

    %% Click Events
    click CLI "https://github.com/xorq-labs/xorq/blob/main/python/xorq/cli.py"
    click API "https://github.com/xorq-labs/xorq/blob/main/python/xorq/expr/api.py"
    click Expr "https://github.com/xorq-labs/xorq/tree/main/python/xorq/expr/"
    click YAML "https://github.com/xorq-labs/xorq/blob/main/python/xorq/ibis_yaml/compiler.py"
    click SQL "https://github.com/xorq-labs/xorq/blob/main/python/xorq/ibis_yaml/sql.py"
    click DF "https://github.com/xorq-labs/xorq/tree/main/python/xorq/backends/datafusion/"
    click DUK "https://github.com/xorq-labs/xorq/tree/main/python/xorq/backends/duckdb/"
    click PG "https://github.com/xorq-labs/xorq/tree/main/python/xorq/backends/postgres/"
    click SF "https://github.com/xorq-labs/xorq/tree/main/python/xorq/backends/snowflake/"
    click TRI "https://github.com/xorq-labs/xorq/tree/main/python/xorq/backends/trino/"
    click CACH "https://github.com/xorq-labs/xorq/tree/main/python/xorq/caching/"
    click UDF "https://github.com/xorq-labs/xorq/blob/main/python/xorq/udf.py"
    click RUST "https://github.com/xorq-labs/xorq/tree/main/src/"
    click EXT "https://github.com/xorq-labs/xorq/tree/main/docker/"
    click DOC "https://github.com/xorq-labs/xorq/tree/main/docs/"
```
