# arvesx/retrievvy architecture

> Architecture diagram of the GitHub repository arvesx/retrievvy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arvesx/retrievvy
- Repository: https://github.com/arvesx/retrievvy

## Overview

##### Project Type and Purpose
- Retrievvy is an open-source retrieval microservice (no GUI/front-end) that exposes HTTP endpoints for bundle ingestion and query.
- It combines “classic” sparse text search (Xapian BM25) and “modern” dense embeddings (FastEmbed + Qdrant) into a single API for fast, reliable document retrieval.

##### File‐Structure Highlights
- Top level: Docker files, env example, scripts, README, `eval/`, and the core package `retrievvy/`.
- `retrievvy/` subpackages:
- `webserver/` – Starlette routes & middleware
- `indexes/` – sparse.py (Xapian), dense.py (Qdrant)
- `nlp/` – embedding and keyword extraction, LLM helpers
- `database.py` – SQLite3 access and schema
- `rerank.py` – fusion of dense+sparse scores
- `chunks.py`, `config.py`, `stats.py`, `utils/` – support code
- `eval/` folder: scripts and datasets for benchmarking metrics.
- `_scripts/pdf_loader.py`: example custom bundle loader.

##### Core Components
a. API Layer
- Starlette/Uvicorn server listening on port 7300
- Routes: `/bundle` (POST ingestion), `/query` (GET search), plus support endpoints (e.g. list indexes)
b. Ingestion Pipeline
1. **Client** sends JSON bundle
2. **Starlette Handler** invokes `retrievvy.chunks` to split bundle → blocks → chunks
3. **Sparse Indexer** (Xapian module) indexes text blocks
4. **Dense Embedder** (FastEmbed via `nlp/embeddings`) → vectors → Qdrant
5. **SQLite** metadata store logs bundle & block IDs, pointers to indexes
c. Query Pipeline
1. **Client** sends query to `/query`
2. **Preprocessor** (tokenize via `tiktoken`, `nltk`)
3. **Sparse Search** → top-n BM25 results
4. **Dense Search** → top-n embeddings results
5. **Reranker** (`retrievvy.rerank`) fuses scores (linear transform, normalization)
6. **Response Serializer** (`msgspec`) → JSON back to client
d. External Services & Libraries
- Qdrant (vector DB)
- Xapian (sparse BM25)
- SQLite3 (lightweight RDBMS)
- msgspec (fast serialization)
- NLTK / yake (keyword extraction)
- FastEmbed (embeddings)
- tenacity (retry logic)

##### Architectural Patterns and Design Principles
- Fully asynchronous, non-blocking API (Starlette + Uvicorn)
- Modular separation by concern: webserver vs. ingestion vs. retrieval vs. eval
- Hybrid retrieval pattern: parallel sparse & dense pipelines with adaptive fusion
- Minimal-footprint design: choose lightweight components by default, but plug-and-play for heavier models

## Diagram (Mermaid)

```mermaid
flowchart TB
    Client["Client"]:::external

    subgraph "API Layer"
        API["Starlette/Uvicorn\nport 7300"]:::api
        Bundles["POST /bundle\nretrievvy/webserver/bundles.py"]:::api
        QueryEP["GET /query\nretrievvy/webserver/vectors.py"]:::api
        IndexesEP["Indexes Endpoint\nretrievvy/webserver/indexes.py"]:::api
        Middleware["Middleware\nretrievvy/webserver/middleware.py"]:::api
        HitsSerializer["Hits Serializer\nmsgspec\nretrievvy/webserver/hits.py"]:::api
    end

    subgraph "Ingestion Pipeline"
        Chunks["Chunking\nretrievvy/chunks.py"]:::core
        SparseIndexer["Sparse Indexer\nXapian\nretrievvy/indexes/sparse.py"]:::sparse
        DenseIndexer["Dense Indexer\nQdrant\nretrievvy/indexes/dense.py"]:::dense
        Embeddings["Embeddings\nFastEmbed\nretrievvy/nlp/embeddings.py"]:::dense
        SQLite["SQLite Metadata Store\nretrievvy/database.py"]:::storage
        Qdrant["Qdrant\nVector DB"]:::storage
        Xapian["Xapian\nText Index"]:::storage
    end

    subgraph "Query Pipeline"
        Keywords["Keyword Extraction\nretrievvy/nlp/keywords.py"]:::core
        LLMPrep["LLM Preprocessor\nretrievvy/nlp/llm.py"]:::core
        SparseSearch["Sparse Search\nBM25\nretrievvy/indexes/sparse.py"]:::sparse
        DenseSearch["Dense Search\nQdrant\nretrievvy/indexes/dense.py"]:::dense
        Reranker["Reranker\nretrievvy/rerank.py"]:::core
    end

    subgraph "Eval & Benchmarking"
        EvalScripts["eval/"]:::aux
    end

    Client -->|HTTP/JSON| API
    API --> Bundles
    API --> QueryEP
    API --> IndexesEP
    Bundles -->|bundle JSON| Chunks
    Chunks --> SparseIndexer
    Chunks --> Embeddings
    Chunks --> DenseIndexer
    SparseIndexer --> Xapian
    DenseIndexer --> Qdrant
    SQLite --> SQLite
    Chunks --> SQLite
    QueryEP --> Keywords
    QueryEP --> LLMPrep
    Keywords --> SparseSearch
    LLMPrep --> DenseSearch
    SparseSearch --> Reranker
    DenseSearch --> Reranker
    Reranker --> HitsSerializer
    HitsSerializer --> Client

    EvalScripts --> API

    click API "https://github.com/arvesx/retrievvy/tree/master/retrievvy/webserver/"
    click Bundles "https://github.com/arvesx/retrievvy/blob/master/retrievvy/webserver/bundles.py"
    click QueryEP "https://github.com/arvesx/retrievvy/blob/master/retrievvy/webserver/vectors.py"
    click IndexesEP "https://github.com/arvesx/retrievvy/blob/master/retrievvy/webserver/indexes.py"
    click Middleware "https://github.com/arvesx/retrievvy/blob/master/retrievvy/webserver/middleware.py"
    click HitsSerializer "https://github.com/arvesx/retrievvy/blob/master/retrievvy/webserver/hits.py"
    click Chunks "https://github.com/arvesx/retrievvy/blob/master/retrievvy/chunks.py"
    click SparseIndexer "https://github.com/arvesx/retrievvy/blob/master/retrievvy/indexes/sparse.py"
    click DenseIndexer "https://github.com/arvesx/retrievvy/blob/master/retrievvy/indexes/dense.py"
    click Embeddings "https://github.com/arvesx/retrievvy/blob/master/retrievvy/nlp/embeddings.py"
    click SQLite "https://github.com/arvesx/retrievvy/blob/master/retrievvy/database.py"
    click Keywords "https://github.com/arvesx/retrievvy/blob/master/retrievvy/nlp/keywords.py"
    click LLMPrep "https://github.com/arvesx/retrievvy/blob/master/retrievvy/nlp/llm.py"
    click Reranker "https://github.com/arvesx/retrievvy/blob/master/retrievvy/rerank.py"
    click EvalScripts "https://github.com/arvesx/retrievvy/tree/master/eval/"

    classDef api fill:#D0E6FF,stroke:#4A90E2,color:#003366
    classDef sparse fill:#E6F5D0,stroke:#6BBF59,color:#3A6010
    classDef dense fill:#FFE5CC,stroke:#FF8C42,color:#663300
    classDef storage fill:#EEEEEE,stroke:#AAAAAA,color:#333333
    classDef core fill:#F0E0FF,stroke:#B366CC,color:#4A0050
    classDef external fill:#F5F5F5,stroke:#BBBBBB,color:#555555
    classDef aux fill:#FFE0E0,stroke:#FF6666,color:#660000
```
