# mooncake-labs/pg_mooncake architecture

> Architecture diagram of the GitHub repository mooncake-labs/pg_mooncake, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mooncake-labs/pg_mooncake
- Repository: https://github.com/mooncake-labs/pg_mooncake

## Overview

##### Main Components
- PostgreSQL Core & Extension Loader
- PostgreSQL as the central DBMS that loads the pg_mooncake extension.
- Columnstore Engine & Handler
- PGDuckDB Integration Module
- Module found in src/pgduckdb, which connects the extension with DuckDB’s vectorized execution engine.
- Components such as “Catalog”, “Planner”, “Scan”, and “Utility” within this module.
- Storage Layer (Lake)
- “Lake” module (from src/lake) that abstracts the underlying storage system. It handles data storage as either Iceberg or Delta Lake tables on the local file system or cloud storage.
- Storage options: “Local FS” and “Cloud Storage (S3/R2)” along with a note that configuration is controlled by parameters (e.g., mooncake.default_bucket).
- Rust Extensions for Delta Lake
- Rust_extensions/delta module which provides support for Delta Lake table operations and builds on the extension’s functionality.
- External and Third-Party Integrations
- A “Third Party” component that encapsulates integrated libraries such as DuckDB (in third_party/duckdb) and additional configuration scripts (e.g., Dockerfiles, Makefiles) that facilitate deployment and testing.
- SQL Scripts & Control Files
- SQL migration scripts in the sql folder and test scripts in the test folder, indicating how database migrations and testing suites interact with the system.

##### Relationships and Interactions
- Data Flow and Control Flow
- From PostgreSQL, responsibilities are handed off to the pg_mooncake extension which routes queries to either the Columnstore Engine or the PGDuckDB module.
- Interactions from the Columnstore Engine to the Lake module for storage operations.
- PGDuckDB module to the third-party DuckDB library—for vectorized query handling.
- That the Rust Delta extension links with the Lake module to support Delta Lake protocols.
- Configuration and Deployment Flow
- A side component showing “Deployment & Tools” (Docker, Makefile, devcontainer) that support building, testing, and deploying the extension across multiple PostgreSQL versions (14–17).

##### Architectural Patterns and Design Principles
- Modular Architecture: Emphasize that the project is divided into distinct modules (extension loader, columnstore, PGDuckDB, lake storage, and Rust-based Delta support) with clearly separated responsibilities.
- Layered Pattern: Express the data flow from the SQL query interface to the execution engine (vectorized processing), and further down to the storage layers.
- Extensibility: The use of both C++ (in the src directories) and Rust (in rust_extensions) illustrates a design that leverages language-specific strengths. Label these modules accordingly.

##### Additional Details
- Interfaces and Boundaries: Show where the system interacts with external systems. For instance, indicate the API boundary between PostgreSQL and pg_mooncake, and between pg_mooncake and external storage services.
- With Key Technologies: Next to each module or connection, briefly note the key framework/libraries (e.g., DuckDB for vectorized execution; Rust for Delta handling).
- Deployment Scenarios: Optionally, indicate how the extension is deployed in Docker or Neon Postgres, demonstrating how configuration files (e.g., Dockerfile, devcontainer configuration) support these deployments.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layer 1: Query Inputs
    subgraph "Layer 1: Query Inputs"
        Client["Client SQL Queries"]:::client
    end

    %% Layer 2: Core Database & Extension
    subgraph "Layer 2: Core Database & Extension"
        PostgreSQLCore["PostgreSQL Core"]:::database
        PgMooncakeExtension["pg_mooncake Extension"]:::database
        SQLScripts["SQL Scripts & Control Files"]:::support
    end

    %% Layer 3: Processing Modules
    subgraph "Layer 3: Processing Modules"
        ColumnstoreEngine["Columnstore Engine"]:::process
        PGDuckDBModule["PGDuckDB Module"]:::process
    end

    %% Layer 4: Storage & Execution
    subgraph "Layer 4: Storage & Execution"
        LakeStorage["Lake Storage (Local FS, Cloud Storage)"]:::storage
        RustDeltaModule["Rust Delta Module"]:::storage
    end

    %% Support & External Integrations
    subgraph "Support & External"
        ThirdParty["Third Party Integrations"]:::external
        DeploymentTools["Deployment & Tools"]:::support
    end

    %% Connections / Data Flow
    Client -->|"sends SQL query"| PostgreSQLCore
    PostgreSQLCore -->|"loads extension"| PgMooncakeExtension
    SQLScripts -->|"run migrations & tests"| PostgreSQLCore
    SQLScripts -->|"validate extension"| PgMooncakeExtension
    PgMooncakeExtension -->|"handles column ops"| ColumnstoreEngine
    PgMooncakeExtension -->|"delegates vector queries"| PGDuckDBModule
    ColumnstoreEngine -->|"performs storage ops"| LakeStorage
    RustDeltaModule -->|"Delta protocol support"| LakeStorage
    PGDuckDBModule -->|"leverages DuckDB"| ThirdParty
    DeploymentTools -->|"deploys system"| PostgreSQLCore
    DeploymentTools -->|"deploys system"| PgMooncakeExtension

    %% Click Events
    click PostgreSQLCore "https://github.com/mooncake-labs/pg_mooncake/blob/main/pg_mooncake.control"
    click PostgreSQLCore "https://github.com/mooncake-labs/pg_mooncake/blob/main/src/pgmooncake.cpp"

    click ColumnstoreEngine "https://github.com/mooncake-labs/pg_mooncake/tree/main/src/columnstore"
    click ColumnstoreEngine "https://github.com/mooncake-labs/pg_mooncake/blob/main/src/columnstore_handler.cpp"

    click PGDuckDBModule "https://github.com/mooncake-labs/pg_mooncake/tree/main/src/pgduckdb"

    click LakeStorage "https://github.com/mooncake-labs/pg_mooncake/tree/main/src/lake"

    click RustDeltaModule "https://github.com/mooncake-labs/pg_mooncake/tree/main/rust_extensions/delta"

    click ThirdParty "https://github.com/mooncake-labs/pg_mooncake/tree/main/third_party"

    click SQLScripts "https://github.com/mooncake-labs/pg_mooncake/tree/main/sql"
    click SQLScripts "https://github.com/mooncake-labs/pg_mooncake/tree/main/test"

    click DeploymentTools "https://github.com/mooncake-labs/pg_mooncake/blob/main/.devcontainer"
    click DeploymentTools "https://github.com/mooncake-labs/pg_mooncake/tree/main/Dockerfile"
    click DeploymentTools "https://github.com/mooncake-labs/pg_mooncake/tree/main/Makefile"

    %% Styles
    classDef client fill:#FFF2CC,stroke:#D4AC0D,stroke-width:2px;
    classDef database fill:#AED6F1,stroke:#2874A6,stroke-width:2px;
    classDef process fill:#D5F5E3,stroke:#28B463,stroke-width:2px;
    classDef storage fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef external fill:#FADBD8,stroke:#C0392B,stroke-width:2px;
    classDef support fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px;
```
