How bazanov05/arxiv-rag-engine works
ArXiv RAG Engine ingests scientific papers, derives lexical signals, learns paper embeddings, stores vectors in PostgreSQL/pgvector, then serves citation-grounded answers through a Gemini tool-calling loop. The interactive CLI initializes the encoder and database pool, delegates each question to the RAG engine, and displays its response and sources. Training and ingestion are separate lifecycle branches supporting the serving index. Several Python modules were only documented or partially sampled, so their internal wiring is shown only where README or excerpts support it.
Main entry points: engine.py RAG engine · main.py Interactive CLI · loader.py Data loader · train.py Training workflow
Components 20
Corpus ingestion
- Paper fetcher
src/data/arxiv_fetcher.py - TF-IDF keywords
tfidf/tfidf.cpp - TF-IDF bindings
tfidf/bindings.cpp - Text normalizer
normalizer/normalizer.cpp - Normalizer bindings
normalizer/bindings.cpp - Data loader
src/data/loader.py
Representation learning
- Training workflow
src/train.py - Triplet dataset
src/data/dataset.py - Triplet loss
src/model/loss.py - Paper encoder
src/model/encoder.py
RAG serving
- Interactive CLI
src/main.py - RAG engine
src/rag/engine.py
Vector storage
- PostgreSQL pgvector
- Database schema
src/db/schema.sql - Connection pool
src/db/connection.py
Other components
- Researcher
- HuggingFace datasets
- ArXiv papers
- Google Gemini API
- Cited answer
Connections 28
- HuggingFace datasets → Paper fetcher: provides papers
- ArXiv papers → Paper fetcher: paper source
- Paper fetcher → Data loader: supplies papers
- Data loader → PostgreSQL pgvector: bulk loads
- TF-IDF bindings → TF-IDF keywords: exposes extractor
- TF-IDF keywords → Data loader: provides keywords
- Normalizer bindings → Text normalizer: exposes cleaner
- Training workflow → Data loader: reads corpus data
- Training workflow → Triplet dataset: builds training data
- Training workflow → Paper encoder: trains projection
- Training workflow → Triplet loss: optimizes with
- Triplet dataset → TF-IDF keywords: uses keyword signals
- Training workflow → PostgreSQL pgvector: reads embeddings
- Database schema → PostgreSQL pgvector: defines tables
- Interactive CLI → Text normalizer: initializes
- Interactive CLI → Paper encoder: loads weights
- Interactive CLI → Connection pool: initializes pool
- Connection pool → PostgreSQL pgvector: connects
- Interactive CLI → RAG engine: submits query
- RAG engine → Text normalizer: cleans query
- RAG engine → Paper encoder: embeds query
- RAG engine → Data loader: requests retrieval
- Data loader → PostgreSQL pgvector: queries vectors
- RAG engine → Google Gemini API: calls agent
- Google Gemini API → RAG engine: dispatches search
- RAG engine → Cited answer: returns grounded text
- Interactive CLI → Cited answer: displays
- Researcher → Interactive CLI: asks question