How lanterndata/lantern works
Project Type and Purpose
• Lantern is primarily a database-centric system:
• A PostgreSQL extension (“lantern_hnsw”) written in C/C++ (via CMake) to provide a high-performance HNSW index on vector columns.
• A Rust-based command-line tool and background daemon (“lantern_cli”) to manage jobs (index building, embeddings, autotuning) and optionally expose a REST API.
• An auxiliary Rust Postgres extension (“lantern_extras”) offering extra search and embedding helper functions (BM25, bloom filters, stemmers, etc.).
• Purpose: let users store vectors in PostgreSQL, generate embeddings (local or via external API), build/serve HNSW indexes (inside or outside the DB), and run nearest-neighbor searches.
File Structure Analysis
• Top-level directories:
• .github, ci, docker: CI/CD, build scripts, Docker-based packaging.
• lantern_hnsw: C library & SQL for the PG extension (usearch integration, HNSW code, SQL update scripts).
• lantern_cli: Rust CLI + daemon + HTTP server + embedding runtimes + client code for external index builder.
• lantern_extras: Rust PGRX extension with helper functions and SQL artifacts.
• Patterns:
• Polyglot codebase (C/C++ for performance, Rust for orchestration and safety).
• Modular design: core indexer (C), CLI/daemon (Rust), extras extension (Rust/PGRX).
• Deployment support: Dockerfiles for CLI, extension, CI workflows, packaging scripts.