# qbeast-io/qbeast-spark architecture

> Architecture diagram of the GitHub repository qbeast-io/qbeast-spark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/qbeast-io/qbeast-spark
- Repository: https://github.com/qbeast-io/qbeast-spark

## Overview

##### Project type and purpose
- This is a multi-module Scala/Java/Spark extension library (an open-source tool) that plugs into Apache Spark to provide a new table format (“qbeast”) on top of Delta Lake.
- Its purpose is to add multi-dimensional indexing, statistical sampling, ACID properties, and optimized I/O into Spark DataFrames/Datasets via a DataSource V2 implementation and SQL extensions.

##### High-level modules (from file tree)
- root (build.sbt, project/, plugins): SBT build definition and global settings
- core/: Pure Scala core model & algorithms (index structures, transformations, metadata management)
- delta/: Delta Lake–specific glue (metadata writer, snapshot integration, Delta transaction hooks)
- src/: Spark extension layer
- internal commands & rules (SQL extension, catalog, DataSource V2)
- catalog and session extension code
- DataSource V2 + relation implementations

##### Main components
a. Spark Application / Driver
- Your user code or spark-shell that registers the Qbeast extension
b. QbeastSparkSessionExtension
- Injects custom analysis & optimization rules (QbeastAnalysis, SampleRule, SaveAsTableRule)
c. QbeastCatalog (Spark SQL catalog extension)
- Provides “CREATE TABLE … USING qbeast”
d. QbeastDataSource V2
- StagedTable (write path)
- BaseRelation / ScanBuilder (read path)
e. Core module
- MetadataManager & IndexManager: maintain and update index files
- OTreeDataAnalyzer / PointWeightIndexer: filter data files by multi-dimensional ranges
- Transformation pipeline: Transformation/Transformer classes → build index structures
f. Delta integration layer
- DeltaMetadataManager / DeltaMetadataWriter: write index changes into the Delta log (Δ-log)
- DeltaQbeastSnapshot: merge Delta Lake snapshots with Qbeast index snapshots
g. Underlying Storage
- Delta Lake transaction log + Parquet data files on HDFS/S3/GCS
h. Python Visualizer (utils/visualizer)
- Reads the same index/Δ-log to generate block/cube visualizations and sampling metrics

##### Interactions and dataflows
- Write flow:
1) DataFrame.write(format="qbeast") → QbeastDataSource.writeStagedTable
2) StagedTable calls Core DataWriter → builds DenormalizedBlock + O-Tree index chunks
3) IndexManager writes index files and registers PreCommitHook
4) DeltaMetadataWriter appends to Δ-log (ACID commit)
- Read flow:
1) DataFrame.read(format="qbeast") → BaseRelation/ScanBuilder
2) QueryExecutor + OTreeDataAnalyzer compute file filters
3) Filter pushdown rules applied in the Spark planner
4) Spark reads only the selected Parquet data files
- Sampling flow:
- qbeastDF.sample(fraction) → SampleRule translates sampling into index range filters → same path as read flow

##### Architectural patterns & design principles
- Spark plugin architecture: session extensions + catalog extension + DataSource V2
- Separation of concerns:
- core (algorithm + model)
- delta (storage integration)
- spark (SQL integration)
- visualizer (external tooling)
- Immutable snapshot + incremental commit (Delta Lake ACID)
- CQRS-style read/write APIs in DataSource V2

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Spark Layer
    subgraph "Spark Layer"
        App["Spark Application / Driver"]:::spark
        Ext["QbeastSparkSessionExtension"]:::spark
        Cat["QbeastCatalog"]:::spark
        subgraph "DataSource V2"
            DS["QbeastDataSource V2"]:::spark
            ST["StagedTable"]:::spark
            BR["BaseRelation / ScanBuilder"]:::spark
            WB["WriteBuilder"]:::spark
            TI["TableImpl"]:::spark
        end
    end

    %% Core Module
    subgraph "Core Module"
        MM["MetadataManager"]:::core
        IM["IndexManager"]:::core
        PWI["PointWeightIndexer"]:::core
        DB["DenormalizedBlock / DataWriter"]:::core
        TP["Transformations Pipeline"]:::core
    end

    %% Spark-side Index Analyzer
    subgraph "Spark-side Index Analyzer"
        ODA["OTreeDataAnalyzer"]:::core
        QE["QueryExecutor"]:::core
        QF["QueryFiltersUtils"]:::core
    end

    %% Delta Integration
    subgraph "Delta Integration"
        DMM["DeltaMetadataManager"]:::delta
        DMW["DeltaMetadataWriter"]:::delta
        DQS["DeltaQbeastSnapshot"]:::delta
        DRH["DeltaRollupDataWriter (hook)"]:::delta
    end

    %% Storage Layer
    Storage["Delta Log + Parquet Files"]:::storage

    %% External Visualizer
    Viz["Python Visualizer (qviz)"]:::external

    %% Connections - Spark setup
    App -->|"register extension"| Ext
    Ext -->|"inject rules"| DS
    Ext -->|"inject rules"| ODA
    Ext -->|"inject rules"| QE

    %% Catalog and DataSource
    Cat -->|"CREATE TABLE USING qbeast"| DS
    DS -->|"writeStagedTable"| DB
    DS -->|"scan"| BR
    DS -->|"scan"| ODA
    DS -->|"sample pushdown"| ODA

    %% Core write flow
    DB -->|"build index"| IM
    IM -->|"update metadata"| MM
    MM -->|"prepare index files"| DMM

    %% Delta commit
    DMM -->|"append Δ-log"| DMW
    DMW -->|"Δ-log commit"| Storage
    DRH -->|"PreCommitHook"| DMW
    DQS -->|"merge snapshots"| Storage

    %% Read flow
    ODA -->|"filter files"| Storage
    QE -->|"apply filters"| BR
    BR -->|"read Parquet"| Storage

    %% External visualizer
    Viz -->|"reads index & Δ-log"| Storage
    Viz -->|"reads index structures"| IM

    %% Click Events
    click Ext "https://github.com/qbeast-io/qbeast-spark/blob/main/src/main/scala/io/qbeast/sql/QbeastSparkSessionExtension.scala"
    click Cat "https://github.com/qbeast-io/qbeast-spark/blob/main/src/main/scala/io/qbeast/catalog/QbeastCatalog.scala"
    click DS "https://github.com/qbeast-io/qbeast-spark/blob/main/src/main/scala/io/qbeast/sources/QbeastDataSource.scala"
    click ST "https://github.com/qbeast-io/qbeast-spark/blob/main/src/main/scala/io/qbeast/sources/v2/QbeastStagedTableImpl.scala"
    click BR "https://github.com/qbeast-io/qbeast-spark/blob/main/src/main/scala/io/qbeast/sources/QbeastBaseRelation.scala"
    click WB "https://github.com/qbeast-io/qbeast-spark/blob/main/src/main/scala/io/qbeast/sources/v2/QbeastWriteBuilder.scala"
    click TI "https://github.com/qbeast-io/qbeast-spark/blob/main/src/main/scala/io/qbeast/sources/v2/QbeastTableImpl.scala"
    click MM "https://github.com/qbeast-io/qbeast-spark/blob/main/core/src/main/scala/io/qbeast/core/model/MetadataManager.scala"
    click IM "https://github.com/qbeast-io/qbeast-spark/blob/main/core/src/main/scala/io/qbeast/core/model/IndexManager.scala"
    click PWI "https://github.com/qbeast-io/qbeast-spark/blob/main/core/src/main/scala/io/qbeast/core/model/PointWeightIndexer.scala"
    click DB "https://github.com/qbeast-io/qbeast-spark/blob/main/core/src/main/scala/io/qbeast/core/model/DenormalizedBlock.scala"
    click DB "https://github.com/qbeast-io/qbeast-spark/blob/main/core/src/main/scala/io/qbeast/core/model/DataWriter.scala"
    click TP "https://github.com/qbeast-io/qbeast-spark/blob/main/core/src/main/scala/io/qbeast/core/transform/*.scala"
    click ODA "https://github.com/qbeast-io/qbeast-spark/blob/main/core/src/main/scala/io/qbeast/spark/index/OTreeDataAnalyzer.scala"
    click QE "https://github.com/qbeast-io/qbeast-spark/blob/main/core/src/main/scala/io/qbeast/spark/index/query/QueryExecutor.scala"
    click QF "https://github.com/qbeast-io/qbeast-spark/blob/main/core/src/main/scala/io/qbeast/spark/index/query/QueryFiltersUtils.scala"
    click DMM "https://github.com/qbeast-io/qbeast-spark/blob/main/delta/src/main/scala/io/qbeast/spark/delta/DeltaMetadataManager.scala"
    click DMW "https://github.com/qbeast-io/qbeast-spark/blob/main/delta/src/main/scala/io/qbeast/spark/delta/DeltaMetadataWriter.scala"
    click DQS "https://github.com/qbeast-io/qbeast-spark/blob/main/delta/src/main/scala/io/qbeast/spark/delta/DeltaQbeastSnapshot.scala"
    click DRH "https://github.com/qbeast-io/qbeast-spark/blob/main/delta/src/main/scala/io/qbeast/spark/delta/DeltaRollupDataWriter.scala"
    click Viz "https://github.com/qbeast-io/qbeast-spark/tree/main/utils/visualizer/qviz/"

    %% Styles
    classDef spark fill:#e0f7fa,stroke:#006064,color:#006064;
    classDef core fill:#fff3e0,stroke:#e65100,color:#e65100;
    classDef delta fill:#e8f5e9,stroke:#2e7d32,color:#2e7d32;
    classDef storage fill:#eceff1,stroke:#455a64,color:#455a64;
    classDef external fill:#f3e5f5,stroke:#6a1b9a,color:#6a1b9a;
```
