# juji-io/datalevin architecture

> Architecture diagram of the GitHub repository juji-io/datalevin, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/juji-io/datalevin
- Repository: https://github.com/juji-io/datalevin

## Overview

##### Project type and purpose
- Datalevin is an embeddable and networked Datalog database library written in Clojure, with native bindings to LMDB (via Java/C++), a cost-based query optimizer, full-text search, and vector search capabilities.
- It can be used as:
- A Clojure library (embedded mode)
- A networked server (default TCP port 8898, RBAC enforced)
- A key-value store for EDN data
- A Babashka pod for scripting

##### Main architectural components
a. Clients & Interfaces
- Clojure API (datalevin.core)
- Babashka pod interface
- Native CLI / Native image (GraalVM)
- Network clients (dtlv:// URI, drivers in other languages planned)
b. Server Component (optional)
- Listener on TCP port 8898
- Authentication & RBAC module
- RPC layer (serialize/deserialize EDN over the wire)
c. Query Layer
- Datalog Parser & Analyzer (src/datalevin/parser.clj, analyzer.clj)
- Query Optimizer (cost-based, chooses index order)
- Query Execution Engine (joins, rule evaluation, pull API)
- Clojure-level helpers (search, full-text, vector)
d. Storage Layer
- LMDB native database (via JNI/C++ under src/java/datalevin/cpp)
- Sub-databases (“DBIs”) for datoms, indices, KV tables
- Buffer management, spill to disk logic
- Asynchronous transaction queue (async.clj)
e. Indexing Extensions
- Full-text search engine (T-Wand integration, search.clj)
- Vector search integration (Usearch library via native-image)
f. Utilities & Tooling
- Schema management (schema-on-write, update-schema)
- CSV/EDN import-export
- Benchmark suites, examples, scripts for native builds

##### Relationships and data flows
- Client issues transact/query calls → API layer routes to Query or KV API
- Query layer parses & optimizes → Execution engine reads/writes via Storage API
- Storage API invokes JNI/C++ wrappers → LMDB performs durable reads/writes
- Async transaction module batches writes, acknowledges clients
- Extensions plug into query/execution layer for full-text and vector search
- In server mode, the network listener converts RPC requests into API calls

##### Architectural patterns & design principles
- Layered architecture: Interface → Query/Execution → Storage
- Pluggable extensions: full-text and vector indexing modules conform to storage/query interfaces
- Schema-on-write: optional schema maps guide storage/indexing
- ACID transactions via LMDB guarantees durability and consistency
- Asynchronous write batching improves throughput under high write loads
- Native-image build for zero-dependency deployment

##### Technologies & frameworks
- Clojure (core logic, API, high-level orchestration)
- Java/C++ JNI wrappers for LMDB (src/java/datalevin/cpp)
- LMDB (Lightning Memory-Mapped Database)
- GraalVM native-image (produces standalone binaries and CLI)
- Usearch (SIMD-accelerated vector search library)
- Babashka pod support
- EDN data format

##### Detailed elements
- Arrowheads indicating call direction or data flow

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Clients Layer
    subgraph "Clients & Interfaces"
        direction TB
        ClojureAPI["Clojure API"]:::client
        BabashkaPod["Babashka Pod"]:::client
        NativeCLI["Native CLI"]:::client
        subgraph "Network Clients"
            direction TB
            RemoteDriver["Network Driver"]:::client
        end
    end

    %% Optional Server Mode
    subgraph "Server Mode" 
        style ServerMode stroke-dasharray: 5 5
        Listener["Network Listener + RBAC"]:::client
        RPCLayer["RPC Layer (EDN RPC)"]:::client
    end

    %% Core Logic
    subgraph "Datalevin Core" 
        direction TB
        Parser["Datalog Parser"]:::core
        Analyzer["Analyzer"]:::core
        Optimizer["Cost-Based Optimizer"]:::core
        ExecutionEngine["Execution Engine"]:::core
        AsyncLayer["Async Transaction Layer"]:::core
        StorageAPI["Storage API & Buffer Management"]:::core
        subgraph "Extensions"
            direction TB
            FullText["Full-Text Search"]:::ext
            VectorSearch["Vector Search"]:::ext
        end
    end

    %% Native & Storage
    subgraph "Native Bindings & Storage"
        direction TB
        JNIWrapper["JNI/C++ Wrapper"]:::native
        LMDB["LMDB Environment\n(datoms, indices, KV tables)"]:::native
    end

    %% Relationships
    ClojureAPI -->|"d/q, d/transact!"| Parser
    BabashkaPod -->|"pod RPC"| Parser
    NativeCLI -->|CLI calls| Parser
    RemoteDriver -->|"dtlv:// RPC"| Listener

    Listener -->|"RPC request"| RPCLayer
    RPCLayer -->|"calls API"| Parser

    Parser -->|"parsed query"| Analyzer
    Analyzer -->|"analyzed query"| Optimizer
    Optimizer -->|"optimized plan"| ExecutionEngine
    ExecutionEngine -->|"exec calls"| StorageAPI
    AsyncLayer -->|"batch writes"| StorageAPI
    StorageAPI -->|"JNI calls"| JNIWrapper
    JNIWrapper -->|"LMDB API"| LMDB

    ExecutionEngine -->|"full-text hooks"| FullText
    ExecutionEngine -->|"vector hooks"| VectorSearch

    %% Click Events
    click ClojureAPI "https://github.com/juji-io/datalevin/blob/master/src/datalevin/core.clj"
    click BabashkaPod "https://github.com/juji-io/datalevin/blob/master/src/pod/huahaiy/datalevin.clj"
    click NativeCLI "https://github.com/juji-io/datalevin/blob/master/src/datalevin/main.clj"
    click Listener "https://github.com/juji-io/datalevin/blob/master/src/datalevin/server.clj"
    click RPCLayer "https://github.com/juji-io/datalevin/blob/master/src/datalevin/protocol.clj"
    click RemoteDriver "https://github.com/juji-io/datalevin/blob/master/src/datalevin/remote.clj"
    click Parser "https://github.com/juji-io/datalevin/blob/master/src/datalevin/parser.clj"
    click Analyzer "https://github.com/juji-io/datalevin/blob/master/src/datalevin/analyzer.clj"
    click Optimizer "https://github.com/juji-io/datalevin/blob/master/src/datalevin/query.clj"
    click ExecutionEngine "https://github.com/juji-io/datalevin/blob/master/src/datalevin/interpret.clj"
    click AsyncLayer "https://github.com/juji-io/datalevin/blob/master/src/datalevin/async.clj"
    click StorageAPI "https://github.com/juji-io/datalevin/blob/master/src/datalevin/storage.clj"
    click JNIWrapper "https://github.com/juji-io/datalevin/blob/master/src/datalevin/lmdb.clj"
    click FullText "https://github.com/juji-io/datalevin/blob/master/src/datalevin/search.clj"
    click VectorSearch "https://github.com/juji-io/datalevin/blob/master/src/datalevin/vector.clj"

    %% Styles
    classDef client fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef core fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef native fill:#FFA500,stroke:#000,stroke-width:1px;
    classDef ext fill:#DDA0DD,stroke:#000,stroke-width:1px;
```
