# lance-format/lance architecture

> Architecture diagram of the GitHub repository lance-format/lance, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lance-format/lance
- Repository: https://github.com/lance-format/lance

## Overview

Lance is a multimodal lakehouse format and SDK for writing, scanning, indexing, and managing versioned datasets. The main workflow is to ingest Arrow-compatible data, persist it as a Lance dataset, then scan or search it; dataset management also includes transactions, schema evolution, and versioned references. The graph combines README-documented capabilities with sampled Python bindings and Rust index internals. Several core implementation files are listed in the SOURCE INDEX but were not excerpted, so unverified internal links are omitted.

Main entry points: [`python/src/dataset.rs`](https://github.com/lance-format/lance/blob/main/python/src/dataset.rs) (Dataset API), [`rust/lance-index/src/vector.rs`](https://github.com/lance-format/lance/blob/main/rust/lance-index/src/vector.rs) (Vector Search), [`python/src/file.rs`](https://github.com/lance-format/lance/blob/main/python/src/file.rs) (Python File API), [`rust/lance/src/dataset`](https://github.com/lance-format/lance/tree/main/rust/lance/src/dataset) (Dataset State)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sdk["SDKs"]
  node_python_sdk["Python SDK"]
  node_java_sdk["Java SDK"]
end

subgraph group_dataset["Dataset Runtime"]
  node_dataset_api["Dataset API<br/>[dataset.rs]"]
  node_dataset_state["Dataset State"]
  node_transactions["Transactions"]
  node_schema_evolution["Schema Evolution"]
  node_namespace["Catalog Namespace<br/>[namespace.rs]"]
  node_scanner["Dataset Scanner<br/>[scanner.rs]"]
end

subgraph group_search["Search and Indexing"]
  node_scalar_search["Scalar and Full-text<br/>[scalar.rs]"]
  node_vector_search["Vector Search<br/>[vector.rs]"]
  node_graph_search["Graph Search<br/>[graph.rs]"]
  node_vector_storage["Vector Index Storage<br/>[storage.rs]"]
  node_quantizers["Quantization<br/>[quantizer.rs]"]
end

subgraph group_format["Storage Format"]
  node_file_api["Python File API<br/>[file.rs]"]
  node_file_reader["Lance File Reader<br/>[reader.rs]"]
  node_file_writer["Lance File Writer<br/>[writer.rs]"]
  node_object_store["Object Storage I/O<br/>[object_store.rs]"]
  node_fragment["Fragment Metadata<br/>[fragment.rs]"]
  node_storage[("Lance Dataset")]
end

node_user(("Data Application"))

node_user -->|"uses"| node_python_sdk
node_python_sdk -.->|"calls"| node_dataset_api
node_user -.->|"uses"| node_java_sdk
node_dataset_api -->|"creates scans"| node_scanner
node_dataset_api -->|"uses"| node_namespace
node_dataset_api -.->|"commits changes"| node_transactions
node_dataset_api -.->|"evolves schema"| node_schema_evolution
node_dataset_api -.->|"supports text search"| node_scalar_search
node_dataset_api -.->|"supports vector search"| node_vector_search
node_scanner -.->|"reads dataset"| node_storage
node_vector_search -->|"searches graph"| node_graph_search
node_vector_search -->|"uses storage"| node_vector_storage
node_vector_storage -->|"loads quantization"| node_quantizers
node_dataset_api -->|"exposes file operations"| node_file_api
node_file_api -.->|"reads files"| node_file_reader
node_file_api -.->|"writes files"| node_file_writer
node_file_reader -.->|"reads storage"| node_object_store
node_file_writer -.->|"writes storage"| node_object_store
node_dataset_state -.->|"uses fragments"| node_fragment
node_dataset_state -.->|"manages versions"| node_storage
node_user -->|"writes dataset"| node_storage
node_user -->|"reads dataset"| node_storage

click node_python_sdk "https://github.com/lance-format/lance/tree/main/python/python/lance"
click node_java_sdk "https://github.com/lance-format/lance/tree/main/java/src/main/java/org/lance"
click node_dataset_api "https://github.com/lance-format/lance/blob/main/python/src/dataset.rs"
click node_dataset_state "https://github.com/lance-format/lance/tree/main/rust/lance/src/dataset"
click node_schema_evolution "https://github.com/lance-format/lance/tree/main/rust/lance/src/dataset"
click node_namespace "https://github.com/lance-format/lance/blob/main/python/src/namespace.rs"
click node_scanner "https://github.com/lance-format/lance/blob/main/python/src/scanner.rs"
click node_scalar_search "https://github.com/lance-format/lance/blob/main/rust/lance-index/src/scalar.rs"
click node_vector_search "https://github.com/lance-format/lance/blob/main/rust/lance-index/src/vector.rs"
click node_graph_search "https://github.com/lance-format/lance/blob/main/rust/lance-index/src/vector/graph.rs"
click node_vector_storage "https://github.com/lance-format/lance/blob/main/rust/lance-index/src/vector/storage.rs"
click node_quantizers "https://github.com/lance-format/lance/blob/main/rust/lance-index/src/vector/quantizer.rs"
click node_file_api "https://github.com/lance-format/lance/blob/main/python/src/file.rs"
click node_file_reader "https://github.com/lance-format/lance/blob/main/rust/lance-file/src/reader.rs"
click node_file_writer "https://github.com/lance-format/lance/blob/main/rust/lance-file/src/writer.rs"
click node_object_store "https://github.com/lance-format/lance/blob/main/rust/lance-io/src/object_store.rs"
click node_fragment "https://github.com/lance-format/lance/blob/main/rust/lance-table/src/format/fragment.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_python_sdk,node_java_sdk toneBlue
class node_dataset_api,node_dataset_state,node_transactions,node_schema_evolution,node_namespace,node_scanner toneAmber
class node_scalar_search,node_vector_search,node_graph_search,node_vector_storage,node_quantizers toneMint
class node_file_api,node_file_reader,node_file_writer,node_object_store,node_fragment,node_storage toneRose
class node_user toneIndigo
```

## Components

### SDKs

- [Python SDK](https://github.com/lance-format/lance/tree/main/python/python/lance): `python/python/lance`
- [Java SDK](https://github.com/lance-format/lance/tree/main/java/src/main/java/org/lance): `java/src/main/java/org/lance`

### Dataset Runtime

- [Dataset API](https://github.com/lance-format/lance/blob/main/python/src/dataset.rs): `python/src/dataset.rs`
- [Dataset State](https://github.com/lance-format/lance/tree/main/rust/lance/src/dataset): `rust/lance/src/dataset`
- **Transactions**
- [Schema Evolution](https://github.com/lance-format/lance/tree/main/rust/lance/src/dataset): `rust/lance/src/dataset`
- [Catalog Namespace](https://github.com/lance-format/lance/blob/main/python/src/namespace.rs): `python/src/namespace.rs`
- [Dataset Scanner](https://github.com/lance-format/lance/blob/main/python/src/scanner.rs): `python/src/scanner.rs`

### Search and Indexing

- [Scalar and Full-text](https://github.com/lance-format/lance/blob/main/rust/lance-index/src/scalar.rs): `rust/lance-index/src/scalar.rs`
- [Vector Search](https://github.com/lance-format/lance/blob/main/rust/lance-index/src/vector.rs): `rust/lance-index/src/vector.rs`
- [Graph Search](https://github.com/lance-format/lance/blob/main/rust/lance-index/src/vector/graph.rs): `rust/lance-index/src/vector/graph.rs`
- [Vector Index Storage](https://github.com/lance-format/lance/blob/main/rust/lance-index/src/vector/storage.rs): `rust/lance-index/src/vector/storage.rs`
- [Quantization](https://github.com/lance-format/lance/blob/main/rust/lance-index/src/vector/quantizer.rs): `rust/lance-index/src/vector/quantizer.rs`

### Storage Format

- [Python File API](https://github.com/lance-format/lance/blob/main/python/src/file.rs): `python/src/file.rs`
- [Lance File Reader](https://github.com/lance-format/lance/blob/main/rust/lance-file/src/reader.rs): `rust/lance-file/src/reader.rs`
- [Lance File Writer](https://github.com/lance-format/lance/blob/main/rust/lance-file/src/writer.rs): `rust/lance-file/src/writer.rs`
- [Object Storage I/O](https://github.com/lance-format/lance/blob/main/rust/lance-io/src/object_store.rs): `rust/lance-io/src/object_store.rs`
- [Fragment Metadata](https://github.com/lance-format/lance/blob/main/rust/lance-table/src/format/fragment.rs): `rust/lance-table/src/format/fragment.rs`
- **Lance Dataset**

### Other components

- **Data Application**

## Connections

- Data Application → Python SDK: uses (evidence: `README.md`)
- Python SDK → Dataset API: calls
- Data Application → Java SDK: uses
- Dataset API → Dataset Scanner: creates scans (evidence: `python/src/dataset.rs`)
- Dataset API → Catalog Namespace: uses (evidence: `python/src/dataset.rs`)
- Dataset API → Transactions: commits changes
- Dataset API → Schema Evolution: evolves schema
- Dataset API → Scalar and Full-text: supports text search (evidence: `README.md`)
- Dataset API → Vector Search: supports vector search (evidence: `README.md`)
- Dataset Scanner → Lance Dataset: reads dataset (evidence: `README.md`)
- Vector Search → Graph Search: searches graph (evidence: `rust/lance-index/src/vector/graph.rs`)
- Vector Search → Vector Index Storage: uses storage (evidence: `rust/lance-index/src/vector/storage.rs`)
- Vector Index Storage → Quantization: loads quantization (evidence: `rust/lance-index/src/vector/storage.rs`)
- Dataset API → Python File API: exposes file operations (evidence: `python/src/dataset.rs`)
- Python File API → Lance File Reader: reads files
- Python File API → Lance File Writer: writes files
- Lance File Reader → Object Storage I/O: reads storage
- Lance File Writer → Object Storage I/O: writes storage
- Dataset State → Fragment Metadata: uses fragments
- Dataset State → Lance Dataset: manages versions (evidence: `README.md`)
- Data Application → Lance Dataset: writes dataset (evidence: `README.md`)
- Data Application → Lance Dataset: reads dataset (evidence: `README.md`)
