How lance-format/lance works
Lance is a multimodal lakehouse format and SDK for writing, scanning, indexing, and managing versioned datasets. The main workflow is to ingest Arrow-compatible data, persist it as a Lance dataset, then scan or search it; dataset management also includes transactions, schema evolution, and versioned references. The graph combines README-documented capabilities with sampled Python bindings and Rust index internals. Several core implementation files are listed in the SOURCE INDEX but were not excerpted, so unverified internal links are omitted.
Main entry points: dataset.rs Dataset API · vector.rs Vector Search · file.rs Python File API · dataset Dataset State
Components 20
SDKs
- Python SDK
python/python/lance - Java SDK
java/src/main/java/org/lance
Dataset Runtime
- Dataset API
python/src/dataset.rs - Dataset State
rust/lance/src/dataset - Transactions
- Schema Evolution
rust/lance/src/dataset - Catalog Namespace
python/src/namespace.rs - Dataset Scanner
python/src/scanner.rs
Search and Indexing
- Scalar and Full-text
rust/lance-index/src/scalar.rs - Vector Search
rust/lance-index/src/vector.rs - Graph Search
rust/lance-index/src/vector/graph.rs - Vector Index Storage
rust/lance-index/src/vector/storage.rs - Quantization
rust/lance-index/src/vector/quantizer.rs
Storage Format
- Python File API
python/src/file.rs - Lance File Reader
rust/lance-file/src/reader.rs - Lance File Writer
rust/lance-file/src/writer.rs - Object Storage I/O
rust/lance-io/src/object_store.rs - Fragment Metadata
rust/lance-table/src/format/fragment.rs - Lance Dataset
Other components
- Data Application
Connections 22
- Data Application → Python SDK: uses
README.md - Python SDK → Dataset API: calls
- Data Application → Java SDK: uses
- Dataset API → Dataset Scanner: creates scans
python/src/dataset.rs - Dataset API → Catalog Namespace: uses
python/src/dataset.rs - Dataset API → Transactions: commits changes
- Dataset API → Schema Evolution: evolves schema
- Dataset API → Scalar and Full-text: supports text search
README.md - Dataset API → Vector Search: supports vector search
README.md - Dataset Scanner → Lance Dataset: reads dataset
README.md - Vector Search → Graph Search: searches graph
rust/lance-index/src/vector/graph.rs - Vector Search → Vector Index Storage: uses storage
rust/lance-index/src/vector/storage.rs - Vector Index Storage → Quantization: loads quantization
rust/lance-index/src/vector/storage.rs - Dataset API → Python File API: exposes file operations
python/src/dataset.rs - Python File API → Lance File Reader: reads files
- Python File API → Lance File Writer: writes files
- Lance File Reader → Object Storage I/O: reads storage
- Lance File Writer → Object Storage I/O: writes storage
- Dataset State → Fragment Metadata: uses fragments
- Dataset State → Lance Dataset: manages versions
README.md - Data Application → Lance Dataset: writes dataset
README.md - Data Application → Lance Dataset: reads dataset
README.md