How qbeast-io/qbeast-spark works
Project type and purpose
• This is a multi-module Scala/Java/Spark extension library (an open-source tool) that plugs into Apache Spark to provide a new table format (“qbeast”) on top of Delta Lake.
• Its purpose is to add multi-dimensional indexing, statistical sampling, ACID properties, and optimized I/O into Spark DataFrames/Datasets via a DataSource V2 implementation and SQL extensions.
High-level modules (from file tree)
• root (build.sbt, project/, plugins): SBT build definition and global settings
• core/: Pure Scala core model & algorithms (index structures, transformations, metadata management)
• delta/: Delta Lake–specific glue (metadata writer, snapshot integration, Delta transaction hooks)
• src/: Spark extension layer
• internal commands & rules (SQL extension, catalog, DataSource V2)
• catalog and session extension code
• DataSource V2 + relation implementations
Main components
a. Spark Application / Driver
• Your user code or spark-shell that registers the Qbeast extension
b. QbeastSparkSessionExtension
• Injects custom analysis & optimization rules (QbeastAnalysis, SampleRule, SaveAsTableRule)
c. QbeastCatalog (Spark SQL catalog extension)
• Provides “CREATE TABLE … USING qbeast”
d. QbeastDataSource V2
• StagedTable (write path)
• BaseRelation / ScanBuilder (read path)
e. Core module
• MetadataManager & IndexManager: maintain and update index files
• OTreeDataAnalyzer / PointWeightIndexer: filter data files by multi-dimensional ranges
• Transformation pipeline: Transformation/Transformer classes → build index structures
f. Delta integration layer
• DeltaMetadataManager / DeltaMetadataWriter: write index changes into the Delta log (Δ-log)
• DeltaQbeastSnapshot: merge Delta Lake snapshots with Qbeast index snapshots
g. Underlying Storage
• Delta Lake transaction log + Parquet data files on HDFS/S3/GCS
h. Python Visualizer (utils/visualizer)
• Reads the same index/Δ-log to generate block/cube visualizations and sampling metrics
Interactions and dataflows
• Write flow:
1) DataFrame.write(format="qbeast") → QbeastDataSource.writeStagedTable
2) StagedTable calls Core DataWriter → builds DenormalizedBlock + O-Tree index chunks
3) IndexManager writes index files and registers PreCommitHook
4) DeltaMetadataWriter appends to Δ-log (ACID commit)
• Read flow:
1) DataFrame.read(format="qbeast") → BaseRelation/ScanBuilder
2) QueryExecutor + OTreeDataAnalyzer compute file filters
3) Filter pushdown rules applied in the Spark planner
4) Spark reads only the selected Parquet data files
• Sampling flow:
• qbeastDF.sample(fraction) → SampleRule translates sampling into index range filters → same path as read flow
Architectural patterns & design principles
• Spark plugin architecture: session extensions + catalog extension + DataSource V2
• Separation of concerns:
• core (algorithm + model)
• delta (storage integration)
• spark (SQL integration)
• visualizer (external tooling)
• Immutable snapshot + incremental commit (Delta Lake ACID)
• CQRS-style read/write APIs in DataSource V2