How scylladb/vector-store works
Project type and purpose
• This is a standalone backend microservice (indexing service) written in Rust.
• Purpose: accept HTTP API calls to index and search vector data, persist metadata in ScyllaDB, and leverage either USearch or OpenSearch for vector search capabilities.
File structure
• src/main.rs + src/lib.rs: application entry point and library glue.
• src/httpserver.rs + src/httproutes.rs: HTTP server and routing layer.
• src/engine.rs: core business logic orchestrator.
• src/db.rs + src/db_index.rs: ScyllaDB connectivity & schema access.
• src/index/: modular index adapters – factory, actor, and two implementations (usearch.rs & opensearch.rs).
• src/monitor_*.rs: background tasks to monitor indexes and items.
• Cargo.toml, Dockerfile: build, dependencies, and containerization.
README for stack and runtime details
• Runs as a Docker container or via cargo run.
• Configurable via environment vars (SCYLLADB_URI, SCYLLA_USEARCH_URI, etc.).
• Exposes HTTP API for clients on configurable port.
Main components and boundaries
a. Clients
– External HTTP clients (e.g., CLI, other services) that issue REST calls.
b. HTTP Server Layer
– httpserver.rs: listens for HTTP requests.
– httproutes.rs: defines endpoints (index, search, health, etc.).
c. Engine Core
– engine.rs: validates requests, applies business rules, and dispatches work.
d. Indexing Subsystem
– factory.rs: selects appropriate adapter based on config (Usearch or OpenSearch).
– actor.rs: runs indexing tasks (possibly as actors/messages).
– usearch.rs / opensearch.rs: adapter implementations that call external vector‐search HTTP APIs.
e. Persistence Layer
– db.rs: handles connection pooling to ScyllaDB.
– db_index.rs: CRUD for index metadata (namespaces, item mappings).
f. Background Monitors
– monitor_indexes.rs & monitor_items.rs: scheduled jobs to sync or rebuild indexes.
g. External Services
– ScyllaDB: vector metadata store.
– USearch or OpenSearch HTTP API: actual vector search engine.
Interactions & data flows
• Client → HTTP Server: POST/GET JSON payloads.
• HTTP Server → Engine: function calls with parsed payloads.
• Engine → Persistence Layer: read/write metadata into Scylla.
• Engine → Indexing Subsystem: push vector data to the chosen external search engine.
• Background Monitors → Persistence + Indexing Subsystem: periodic reconciliation.
• All external calls over TCP/HTTP.
Architectural patterns & principles
• Modular adapters (Factory Pattern) for pluggable backends.
• Layered architecture: API Layer → Business Logic → Persistence/External Integration.
• Concurrency: background tasks decoupled via actor messaging.
• Configuration via environment (12-factor style).
• All major components labeled with module names.
• Annotations for protocols (HTTP, CQL) and config points (env vars, Docker).