How juji-io/datalevin works
Project type and purpose
• Datalevin is an embeddable and networked Datalog database library written in Clojure, with native bindings to LMDB (via Java/C++), a cost-based query optimizer, full-text search, and vector search capabilities.
• It can be used as:
• A Clojure library (embedded mode)
• A networked server (default TCP port 8898, RBAC enforced)
• A key-value store for EDN data
• A Babashka pod for scripting
Main architectural components
a. Clients & Interfaces
• Clojure API (datalevin.core)
• Babashka pod interface
• Native CLI / Native image (GraalVM)
• Network clients (dtlv:// URI, drivers in other languages planned)
b. Server Component (optional)
• Listener on TCP port 8898
• Authentication & RBAC module
• RPC layer (serialize/deserialize EDN over the wire)
c. Query Layer
• Datalog Parser & Analyzer (src/datalevin/parser.clj, analyzer.clj)
• Query Optimizer (cost-based, chooses index order)
• Query Execution Engine (joins, rule evaluation, pull API)
• Clojure-level helpers (search, full-text, vector)
d. Storage Layer
• LMDB native database (via JNI/C++ under src/java/datalevin/cpp)
• Sub-databases (“DBIs”) for datoms, indices, KV tables
• Buffer management, spill to disk logic
• Asynchronous transaction queue (async.clj)
e. Indexing Extensions
• Full-text search engine (T-Wand integration, search.clj)
• Vector search integration (Usearch library via native-image)
f. Utilities & Tooling
• Schema management (schema-on-write, update-schema)
• CSV/EDN import-export
• Benchmark suites, examples, scripts for native builds
Relationships and data flows
• Client issues transact/query calls → API layer routes to Query or KV API
• Query layer parses & optimizes → Execution engine reads/writes via Storage API
• Storage API invokes JNI/C++ wrappers → LMDB performs durable reads/writes
• Async transaction module batches writes, acknowledges clients
• Extensions plug into query/execution layer for full-text and vector search
• In server mode, the network listener converts RPC requests into API calls
Architectural patterns & design principles
• Layered architecture: Interface → Query/Execution → Storage
• Pluggable extensions: full-text and vector indexing modules conform to storage/query interfaces
• Schema-on-write: optional schema maps guide storage/indexing
• ACID transactions via LMDB guarantees durability and consistency
• Asynchronous write batching improves throughput under high write loads
• Native-image build for zero-dependency deployment
Technologies & frameworks
• Clojure (core logic, API, high-level orchestration)
• Java/C++ JNI wrappers for LMDB (src/java/datalevin/cpp)
• LMDB (Lightning Memory-Mapped Database)
• GraalVM native-image (produces standalone binaries and CLI)
• Usearch (SIMD-accelerated vector search library)
• Babashka pod support
• EDN data format
Detailed elements
• Arrowheads indicating call direction or data flow