How lance-format/lance-spark works
This repository is the Apache Spark connector for Lance: it integrates Lance datasets into Spark SQL for reading, searching, writing, and dataset management. The main user path is Spark SQL through registered extensions and planning into Lance-backed scans or writes. Search functions and optimizer rules add specialized query behavior. The README establishes the connector’s read-and-query purpose; sampled extensions and source-index references support the principal integration points. Several core implementations are listed but unsampled, so uncertain internal calls are omitted.
Main entry points: LanceSparkSessionExtensions.scala Session extensions · LanceSearchQuery.java Search query spec · LanceDataSourceV2Strategy.scala V2 planner strategy · LanceDataset.java Lance dataset
Components 23
Spark SQL integration
- Session extensions
lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala - SQL parser
lance-spark-3.4_2.12/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/LanceSparkSqlExtensionsParser.scala - V2 planner strategy
lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/LanceDataSourceV2Strategy.scala - Dataset SQL operations
lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/plans/logical
Dataset access
- Namespace catalog
lance-spark-base_2.12/src/main/java/org/lance/spark/BaseLanceNamespaceSparkCatalog.java - Lance dataset
lance-spark-base_2.12/src/main/java/org/lance/spark/LanceDataset.java - Read options
lance-spark-base_2.12/src/main/java/org/lance/spark/LanceSparkReadOptions.java - Fragment scanner
lance-spark-base_2.12/src/main/java/org/lance/spark/internal/LanceFragmentColumnarBatchScanner.java - Fragment pruning
lance-spark-base_2.12/src/main/java/org/lance/spark/read/ZonemapFragmentPruner.java
Search and query
- Search table functions
lance-spark-base_2.12/src/main/scala/org/lance/spark/search/LanceSearchTableFunctions.scala - Search query spec
lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceSearchQuery.java - Search scans
lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceSearchScan.java - Hybrid search
lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceHybridSearchQuery.java - Full-text conversion
lance-spark-base_2.12/src/main/java/org/lance/spark/utils/FullTextQueryUtils.java
Writes and blobs
- Write operations
lance-spark-base_2.12/src/main/java/org/lance/spark/write/SparkWrite.java - Blob handling
lance-spark-base_2.12/src/main/java/org/lance/spark/utils/BlobUtils.java - Blob copy-through
lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceBlobV2CopyThroughRule.scala - Row-level blob copy
lance-spark-3.5_2.12/src/main/scala-rowlevel/org/apache/spark/sql/catalyst/optimizer/LanceBlobV2RowLevelCopyRule.scala - Blob source context
lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceBlobSourceContextRule.scala
Planning optimizations
- Fragment-aware joins
lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceFragmentAwareJoinRule.scala - FTS predicate rule
lance-spark-3.4_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceFtsPredicateRule.scala
Other components
- Spark SQL user
- Lance datasets
Connections 24
- Spark SQL user → Session extensions: configures
- Session extensions → SQL parser: injects parser
lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala - Session extensions → V2 planner strategy: injects strategy
lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala - Session extensions → Search table functions: registers functions
lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala - Session extensions → Fragment-aware joins: registers rule
lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala - Session extensions → FTS predicate rule: registers rule
lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala - Session extensions → Blob copy-through: registers rule
lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala - Session extensions → Row-level blob copy: registers rule
lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala - Session extensions → Blob source context: registers rule
lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala - Search query spec → Full-text conversion: converts query
lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceSearchQuery.java - Search query spec → Search scans: configures scan
- Search table functions → Search query spec: builds query
lance-spark-base_2.12/src/main/scala/org/lance/spark/search/LanceSearchTableFunctions.scala - Hybrid search → Search query spec: uses query spec
lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceHybridSearchQuery.java - V2 planner strategy → Dataset SQL operations: plans operations
lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/LanceDataSourceV2Strategy.scala - Lance dataset → Read options: exposes options
- Fragment scanner → Lance datasets: reads fragments
- Fragment pruning → Fragment scanner: prunes fragments
- Namespace catalog → Lance dataset: resolves datasets
- Write operations → Lance datasets: writes data
- Blob copy-through → Blob handling: checks blob metadata
lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceBlobV2CopyThroughRule.scala - Row-level blob copy → Blob handling: checks blob metadata
lance-spark-3.5_2.12/src/main/scala-rowlevel/org/apache/spark/sql/catalyst/optimizer/LanceBlobV2RowLevelCopyRule.scala - Blob source context → Lance dataset: reads dataset context
lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceBlobSourceContextRule.scala - FTS predicate rule → Search scans: pushes predicates
- Fragment-aware joins → V2 planner strategy: rewrites joins