# lancedb/lance architecture

> Architecture diagram of the GitHub repository lancedb/lance, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lancedb/lance
- Repository: https://github.com/lancedb/lance

## Overview

##### Project Type and Purpose
- This is an open-source, multi-language library and toolkit (“Lance”) implementing a columnar data format optimized for ML.
- At its heart is a Rust core (crates under `rust/` and `python/src/`, plus sub-crates like `lance-core`, `lance-file`, `lance-encoding`, `lance-index`, `lance-io`, `lance-datagen`, `lance-datafusion`, etc.).
- Language bindings sit on top: Python via PyO3 (`python/`), Java via JNI (`java/core/lance-jni/`), and a Spark connector (`java/spark/`).
- There are docs, benchmarks, examples, notebooks, CI/CD workflows, and protobuf schemas for on-the-wire formats.

##### File Structure & Architectural Patterns
- Top-level modular separation:
- `rust/` – core engine and utilities
- `python/` – Python bindings and test suite
- `java/` – JNI layer & Spark integration
- `protos/` – shared protobuf definitions (schema, index, transactions)
- `docs/`, `benchmarks/`, `notebooks/`, `.github/`, CI files, etc.
- Pattern: “Core + Adapters”
- Core crates implement dataset, file layout, encodings, indices, IO, transactions, versioning.
- Language-specific adapters bind into the core via PyO3 or JNI.
- Spark integration uses the Java adapter to present a DataSource to Spark.
- Core Engine subdivided functionally:
- Data model & schema evolution
- File and manifest layout (read/write)
- Encodings & compression
- Indexing (vector, scalar, text)
- IO & object store (S3, GCS, Azure, local)
- Execution/planner extensions (DataFusion, Spark pushdowns)
- CLI and examples (Rust binaries in `rust/lance/src/bin/`)

##### Essential Elements
- External object stores and file system as storage targets.
- Clients (Python REPL, Spark, Rust CLI, notebooks) consuming the bindings.
- CI/CD and documentation pipelines at the periphery to show how releases and docs are generated.

## Diagram (Mermaid)

```mermaid
flowchart TB
  subgraph "Clients" 
    direction TB
    A1["Python Apps (Pandas, DuckDB, Polars)"]:::external
    A4["Notebooks"]:::external
    A2["Spark Jobs"]:::external
    A3["Rust CLI"]:::external
  end

  subgraph "Language Bindings" 
    direction TB
    B1["Python Binding (PyO3)"]:::bindings
    B2["Java Binding (JNI)"]:::bindings
    B3["Spark Connector"]:::bindings
  end

  subgraph "Protobuf Schemas" 
    P["Protos"]:::shared
  end

  subgraph "Core Engine (Rust)" 
    direction TB
    C1["Dataset API & Transaction Manager"]:::core
    C2["File & Manifest Layout Manager"]:::core
    C3["Encodings & Compression"]:::core
    C4["Index Modules"]:::core
    C5["IO Layer & Object-Store Providers"]:::core
    C6["DataFusion Extensions"]:::core
  end

  subgraph "Storage" 
    direction TB
    S1["Object Stores (S3, GCS, Azure)"]:::external
    S2["Local FS"]:::external
  end

  subgraph "CI/CD & Docs" 
    direction TB
    D1["CI/CD Workflows"]:::shared
    D2["Documentation"]:::shared
  end

  %% Client to Bindings/Core
  A1 -->|"FFI call"| B1
  A4 -->|"FFI call"| B1
  A2 -->|"JNI call"| B2
  A2 -->|"Spark DS API"| B3
  A3 -->|"Rust call"| C1

  %% Bindings to Protobuf
  B1 ---|"gRPC/protobuf"| P
  B2 ---|"gRPC/protobuf"| P

  %% Bindings to Core
  B1 -->|"FFI call"| C1
  B2 -->|"JNI call"| C1
  B3 -->|"Pushdown"| C6
  B3 -->|"API call"| C1

  %% Protobuf to Core
  P ---|"schema definitions"| C1

  %% Core modules flow
  C1 --> C2
  C2 --> C3
  C3 --> C4
  C4 --> C5
  C5 --> C6

  %% Core to Storage
  C5 -->|"HTTP/S3"| S1
  C5 -->|"FS"| S2

  %% Core to Bindings return
  C1 -->|"Arrow batches"| B1
  C1 -->|"Arrow batches"| B2
  C6 -->|"ColumnarBatches"| B3

  %% CI/CD & Docs connections
  D1 -.->|"triggers builds"| C1
  D1 -.->|"deploy docs"| D2

  %% Click Events
  click C1 "https://github.com/lancedb/lance/tree/main/rust/lance/src/dataset"
  click C2 "https://github.com/lancedb/lance/tree/main/rust/lance-file/src"
  click C3 "https://github.com/lancedb/lance/tree/main/rust/lance-encoding/src"
  click C4 "https://github.com/lancedb/lance/tree/main/rust/lance-index/src"
  click C5 "https://github.com/lancedb/lance/tree/main/rust/lance-io/src"
  click C6 "https://github.com/lancedb/lance/tree/main/rust/lance-datafusion/src"
  click A3 "https://github.com/lancedb/lance/tree/main/rust/lance/src/bin"
  click A4 "https://github.com/lancedb/lance/tree/main/rust/lance/examples"
  click B1 "https://github.com/lancedb/lance/tree/main/python/src"
  click B2 "https://github.com/lancedb/lance/tree/main/java/core/lance-jni/src"
  click B3 "https://github.com/lancedb/lance/tree/main/java/spark/src/main/java/com/lancedb/lance/spark"
  click P "https://github.com/lancedb/lance/tree/main/protos/"
  click D2 "https://github.com/lancedb/lance/tree/main/docs/"
  click D1 "https://github.com/lancedb/lance/tree/main/.github/workflows/"

  %% Legend Styles
  classDef core fill:#cce5ff,stroke:#333;
  classDef bindings fill:#d4edda,stroke:#333;
  classDef external fill:#ffe5b4,stroke:#333;
  classDef shared fill:#e2e3e5,stroke:#333;
```
