# lance-format/lance-spark architecture

> Architecture diagram of the GitHub repository lance-format/lance-spark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lance-format/lance-spark
- Repository: https://github.com/lance-format/lance-spark

## Overview

This repository is the Apache Spark connector for Lance: it integrates Lance datasets into Spark SQL for reading, searching, writing, and dataset management. The main user path is Spark SQL through registered extensions and planning into Lance-backed scans or writes. Search functions and optimizer rules add specialized query behavior. The README establishes the connector’s read-and-query purpose; sampled extensions and source-index references support the principal integration points. Several core implementations are listed but unsampled, so uncertain internal calls are omitted.

Main entry points: [`lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala`](https://github.com/lance-format/lance-spark/blob/main/lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala) (Session extensions), [`lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceSearchQuery.java`](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceSearchQuery.java) (Search query spec), [`lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/LanceDataSourceV2Strategy.scala`](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/LanceDataSourceV2Strategy.scala) (V2 planner strategy), [`lance-spark-base_2.12/src/main/java/org/lance/spark/LanceDataset.java`](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/LanceDataset.java) (Lance dataset)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sql["Spark SQL integration"]
  node_extensions["Session extensions"]
  node_parser["SQL parser"]
  node_planner["V2 planner strategy"]
  node_v2_operations["Dataset SQL operations<br/>[logical]"]
end

subgraph group_data["Dataset access"]
  node_catalog["Namespace catalog"]
  node_dataset["Lance dataset<br/>[LanceDataset.java]"]
  node_read_options["Read options"]
  node_fragment_scan["Fragment scanner"]
  node_pruning["Fragment pruning"]
end

subgraph group_search["Search and query"]
  node_search_functions["Search table functions"]
  node_search_query["Search query spec"]
  node_search_scan["Search scans"]
  node_hybrid_query["Hybrid search"]
  node_fts_utils["Full-text conversion"]
end

subgraph group_write["Writes and blobs"]
  node_write_path["Write operations<br/>[SparkWrite.java]"]
  node_blob_utils["Blob handling<br/>[BlobUtils.java]"]
  node_blob_copy["Blob copy-through"]
  node_rowlevel_copy["Row-level blob copy"]
  node_source_context["Blob source context"]
end

subgraph group_opt["Planning optimizations"]
  node_join_rule["Fragment-aware joins"]
  node_fts_rule["FTS predicate rule"]
end

node_spark_user(("Spark SQL user"))
node_lance_store[("Lance datasets")]

node_spark_user -.->|"configures"| node_extensions
node_extensions -->|"injects parser"| node_parser
node_extensions -->|"injects strategy"| node_planner
node_extensions -->|"registers functions"| node_search_functions
node_extensions -->|"registers rule"| node_join_rule
node_extensions -->|"registers rule"| node_fts_rule
node_extensions -->|"registers rule"| node_blob_copy
node_extensions -->|"registers rule"| node_rowlevel_copy
node_extensions -->|"registers rule"| node_source_context
node_search_query -->|"converts query"| node_fts_utils
node_search_query -.->|"configures scan"| node_search_scan
node_search_functions -.->|"builds query"| node_search_query
node_hybrid_query -->|"uses query spec"| node_search_query
node_planner -->|"plans operations"| node_v2_operations
node_dataset -.->|"exposes options"| node_read_options
node_fragment_scan -.->|"reads fragments"| node_lance_store
node_pruning -.->|"prunes fragments"| node_fragment_scan
node_catalog -.->|"resolves datasets"| node_dataset
node_write_path -.->|"writes data"| node_lance_store
node_blob_copy -->|"checks blob metadata"| node_blob_utils
node_rowlevel_copy -->|"checks blob metadata"| node_blob_utils
node_source_context -->|"reads dataset context"| node_dataset
node_fts_rule -.->|"pushes predicates"| node_search_scan
node_join_rule -.->|"rewrites joins"| node_planner

click node_extensions "https://github.com/lance-format/lance-spark/blob/main/lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala"
click node_parser "https://github.com/lance-format/lance-spark/blob/main/lance-spark-3.4_2.12/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/LanceSparkSqlExtensionsParser.scala"
click node_planner "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/LanceDataSourceV2Strategy.scala"
click node_catalog "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/BaseLanceNamespaceSparkCatalog.java"
click node_dataset "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/LanceDataset.java"
click node_read_options "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/LanceSparkReadOptions.java"
click node_search_functions "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/scala/org/lance/spark/search/LanceSearchTableFunctions.scala"
click node_search_query "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceSearchQuery.java"
click node_search_scan "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceSearchScan.java"
click node_hybrid_query "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceHybridSearchQuery.java"
click node_fts_utils "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/utils/FullTextQueryUtils.java"
click node_fragment_scan "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/internal/LanceFragmentColumnarBatchScanner.java"
click node_pruning "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/read/ZonemapFragmentPruner.java"
click node_v2_operations "https://github.com/lance-format/lance-spark/tree/main/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/plans/logical"
click node_write_path "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/write/SparkWrite.java"
click node_blob_utils "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/utils/BlobUtils.java"
click node_blob_copy "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceBlobV2CopyThroughRule.scala"
click node_rowlevel_copy "https://github.com/lance-format/lance-spark/blob/main/lance-spark-3.5_2.12/src/main/scala-rowlevel/org/apache/spark/sql/catalyst/optimizer/LanceBlobV2RowLevelCopyRule.scala"
click node_source_context "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceBlobSourceContextRule.scala"
click node_join_rule "https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceFragmentAwareJoinRule.scala"
click node_fts_rule "https://github.com/lance-format/lance-spark/blob/main/lance-spark-3.4_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceFtsPredicateRule.scala"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_extensions,node_parser,node_planner,node_v2_operations,node_spark_user toneBlue
class node_catalog,node_dataset,node_read_options,node_fragment_scan,node_pruning,node_lance_store toneAmber
class node_search_functions,node_search_query,node_search_scan,node_hybrid_query,node_fts_utils toneMint
class node_write_path,node_blob_utils,node_blob_copy,node_rowlevel_copy,node_source_context toneRose
class node_join_rule,node_fts_rule toneIndigo
```

## Components

### Spark SQL integration

- [Session extensions](https://github.com/lance-format/lance-spark/blob/main/lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala): `lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala`
- [SQL parser](https://github.com/lance-format/lance-spark/blob/main/lance-spark-3.4_2.12/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/LanceSparkSqlExtensionsParser.scala): `lance-spark-3.4_2.12/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/LanceSparkSqlExtensionsParser.scala`
- [V2 planner strategy](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/LanceDataSourceV2Strategy.scala): `lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/LanceDataSourceV2Strategy.scala`
- [Dataset SQL operations](https://github.com/lance-format/lance-spark/tree/main/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/plans/logical): `lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/plans/logical`

### Dataset access

- [Namespace catalog](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/BaseLanceNamespaceSparkCatalog.java): `lance-spark-base_2.12/src/main/java/org/lance/spark/BaseLanceNamespaceSparkCatalog.java`
- [Lance dataset](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/LanceDataset.java): `lance-spark-base_2.12/src/main/java/org/lance/spark/LanceDataset.java`
- [Read options](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/LanceSparkReadOptions.java): `lance-spark-base_2.12/src/main/java/org/lance/spark/LanceSparkReadOptions.java`
- [Fragment scanner](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/internal/LanceFragmentColumnarBatchScanner.java): `lance-spark-base_2.12/src/main/java/org/lance/spark/internal/LanceFragmentColumnarBatchScanner.java`
- [Fragment pruning](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/read/ZonemapFragmentPruner.java): `lance-spark-base_2.12/src/main/java/org/lance/spark/read/ZonemapFragmentPruner.java`

### Search and query

- [Search table functions](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/scala/org/lance/spark/search/LanceSearchTableFunctions.scala): `lance-spark-base_2.12/src/main/scala/org/lance/spark/search/LanceSearchTableFunctions.scala`
- [Search query spec](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceSearchQuery.java): `lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceSearchQuery.java`
- [Search scans](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceSearchScan.java): `lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceSearchScan.java`
- [Hybrid search](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceHybridSearchQuery.java): `lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceHybridSearchQuery.java`
- [Full-text conversion](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/utils/FullTextQueryUtils.java): `lance-spark-base_2.12/src/main/java/org/lance/spark/utils/FullTextQueryUtils.java`

### Writes and blobs

- [Write operations](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/write/SparkWrite.java): `lance-spark-base_2.12/src/main/java/org/lance/spark/write/SparkWrite.java`
- [Blob handling](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/java/org/lance/spark/utils/BlobUtils.java): `lance-spark-base_2.12/src/main/java/org/lance/spark/utils/BlobUtils.java`
- [Blob copy-through](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceBlobV2CopyThroughRule.scala): `lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceBlobV2CopyThroughRule.scala`
- [Row-level blob copy](https://github.com/lance-format/lance-spark/blob/main/lance-spark-3.5_2.12/src/main/scala-rowlevel/org/apache/spark/sql/catalyst/optimizer/LanceBlobV2RowLevelCopyRule.scala): `lance-spark-3.5_2.12/src/main/scala-rowlevel/org/apache/spark/sql/catalyst/optimizer/LanceBlobV2RowLevelCopyRule.scala`
- [Blob source context](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceBlobSourceContextRule.scala): `lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceBlobSourceContextRule.scala`

### Planning optimizations

- [Fragment-aware joins](https://github.com/lance-format/lance-spark/blob/main/lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceFragmentAwareJoinRule.scala): `lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceFragmentAwareJoinRule.scala`
- [FTS predicate rule](https://github.com/lance-format/lance-spark/blob/main/lance-spark-3.4_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceFtsPredicateRule.scala): `lance-spark-3.4_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceFtsPredicateRule.scala`

### Other components

- **Spark SQL user**
- **Lance datasets**

## Connections

- Spark SQL user → Session extensions: configures
- Session extensions → SQL parser: injects parser (evidence: `lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala`)
- Session extensions → V2 planner strategy: injects strategy (evidence: `lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala`)
- Session extensions → Search table functions: registers functions (evidence: `lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala`)
- Session extensions → Fragment-aware joins: registers rule (evidence: `lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala`)
- Session extensions → FTS predicate rule: registers rule (evidence: `lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala`)
- Session extensions → Blob copy-through: registers rule (evidence: `lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala`)
- Session extensions → Row-level blob copy: registers rule (evidence: `lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala`)
- Session extensions → Blob source context: registers rule (evidence: `lance-spark-4.2_2.13/src/main/scala/org/lance/spark/extensions/LanceSparkSessionExtensions.scala`)
- Search query spec → Full-text conversion: converts query (evidence: `lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceSearchQuery.java`)
- Search query spec → Search scans: configures scan
- Search table functions → Search query spec: builds query (evidence: `lance-spark-base_2.12/src/main/scala/org/lance/spark/search/LanceSearchTableFunctions.scala`)
- Hybrid search → Search query spec: uses query spec (evidence: `lance-spark-base_2.12/src/main/java/org/lance/spark/search/LanceHybridSearchQuery.java`)
- V2 planner strategy → Dataset SQL operations: plans operations (evidence: `lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/execution/datasources/v2/LanceDataSourceV2Strategy.scala`)
- Lance dataset → Read options: exposes options
- Fragment scanner → Lance datasets: reads fragments
- Fragment pruning → Fragment scanner: prunes fragments
- Namespace catalog → Lance dataset: resolves datasets
- Write operations → Lance datasets: writes data
- Blob copy-through → Blob handling: checks blob metadata (evidence: `lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceBlobV2CopyThroughRule.scala`)
- Row-level blob copy → Blob handling: checks blob metadata (evidence: `lance-spark-3.5_2.12/src/main/scala-rowlevel/org/apache/spark/sql/catalyst/optimizer/LanceBlobV2RowLevelCopyRule.scala`)
- Blob source context → Lance dataset: reads dataset context (evidence: `lance-spark-base_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LanceBlobSourceContextRule.scala`)
- FTS predicate rule → Search scans: pushes predicates
- Fragment-aware joins → V2 planner strategy: rewrites joins
