How arvesx/retrievvy works
Project Type and Purpose
• Retrievvy is an open-source retrieval microservice (no GUI/front-end) that exposes HTTP endpoints for bundle ingestion and query.
• It combines “classic” sparse text search (Xapian BM25) and “modern” dense embeddings (FastEmbed + Qdrant) into a single API for fast, reliable document retrieval.
File‐Structure Highlights
• Top level: Docker files, env example, scripts, README, eval/, and the core package retrievvy/.
• retrievvy/ subpackages:
• webserver/ – Starlette routes & middleware
• indexes/ – sparse.py (Xapian), dense.py (Qdrant)
• nlp/ – embedding and keyword extraction, LLM helpers
• database.py – SQLite3 access and schema
• rerank.py – fusion of dense+sparse scores
• chunks.py, config.py, stats.py, utils/ – support code
• eval/ folder: scripts and datasets for benchmarking metrics.
• _scripts/pdf_loader.py: example custom bundle loader.
Core Components
a. API Layer
• Starlette/Uvicorn server listening on port 7300
• Routes: /bundle (POST ingestion), /query (GET search), plus support endpoints (e.g. list indexes)
b. Ingestion Pipeline
1. Client sends JSON bundle
2. Starlette Handler invokes retrievvy.chunks to split bundle → blocks → chunks
3. Sparse Indexer (Xapian module) indexes text blocks
4. Dense Embedder (FastEmbed via nlp/embeddings) → vectors → Qdrant
5. SQLite metadata store logs bundle & block IDs, pointers to indexes
c. Query Pipeline
1. Client sends query to /query
2. Preprocessor (tokenize via tiktoken, nltk)
3. Sparse Search → top-n BM25 results
4. Dense Search → top-n embeddings results
5. Reranker (retrievvy.rerank) fuses scores (linear transform, normalization)
6. Response Serializer (msgspec) → JSON back to client
d. External Services & Libraries
• Qdrant (vector DB)
• Xapian (sparse BM25)
• SQLite3 (lightweight RDBMS)
• msgspec (fast serialization)
• NLTK / yake (keyword extraction)
• FastEmbed (embeddings)
• tenacity (retry logic)
Architectural Patterns and Design Principles
• Fully asynchronous, non-blocking API (Starlette + Uvicorn)
• Modular separation by concern: webserver vs. ingestion vs. retrieval vs. eval
• Hybrid retrieval pattern: parallel sparse & dense pipelines with adaptive fusion
• Minimal-footprint design: choose lightweight components by default, but plug-and-play for heavier models