How lancedb/lance works
Project Type and Purpose
• This is an open-source, multi-language library and toolkit (“Lance”) implementing a columnar data format optimized for ML.
• At its heart is a Rust core (crates under rust/ and python/src/, plus sub-crates like lance-core, lance-file, lance-encoding, lance-index, lance-io, lance-datagen, lance-datafusion, etc.).
• Language bindings sit on top: Python via PyO3 (python/), Java via JNI (java/core/lance-jni/), and a Spark connector (java/spark/).
• There are docs, benchmarks, examples, notebooks, CI/CD workflows, and protobuf schemas for on-the-wire formats.
File Structure & Architectural Patterns
• Top-level modular separation:
• rust/ – core engine and utilities
• python/ – Python bindings and test suite
• java/ – JNI layer & Spark integration
• protos/ – shared protobuf definitions (schema, index, transactions)
• docs/, benchmarks/, notebooks/, .github/, CI files, etc.
• Pattern: “Core + Adapters”
• Core crates implement dataset, file layout, encodings, indices, IO, transactions, versioning.
• Language-specific adapters bind into the core via PyO3 or JNI.
• Spark integration uses the Java adapter to present a DataSource to Spark.
• Core Engine subdivided functionally:
• Data model & schema evolution
• File and manifest layout (read/write)
• Encodings & compression
• Indexing (vector, scalar, text)
• IO & object store (S3, GCS, Azure, local)
• Execution/planner extensions (DataFusion, Spark pushdowns)
• CLI and examples (Rust binaries in rust/lance/src/bin/)
Essential Elements
• External object stores and file system as storage targets.
• Clients (Python REPL, Spark, Rust CLI, notebooks) consuming the bindings.
• CI/CD and documentation pipelines at the periphery to show how releases and docs are generated.