# bazanov05/arxiv-rag-engine architecture

> Architecture diagram of the GitHub repository bazanov05/arxiv-rag-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bazanov05/arxiv-rag-engine
- Repository: https://github.com/bazanov05/arxiv-rag-engine
- Video tour (about a minute): https://gitdiagram.com/bazanov05/arxiv-rag-engine/video

## Overview

ArXiv RAG Engine ingests scientific papers, derives lexical signals, learns paper embeddings, stores vectors in PostgreSQL/pgvector, then serves citation-grounded answers through a Gemini tool-calling loop. The interactive CLI initializes the encoder and database pool, delegates each question to the RAG engine, and displays its response and sources. Training and ingestion are separate lifecycle branches supporting the serving index. Several Python modules were only documented or partially sampled, so their internal wiring is shown only where README or excerpts support it.

Main entry points: [`src/rag/engine.py`](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/rag/engine.py) (RAG engine), [`src/main.py`](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/main.py) (Interactive CLI), [`src/data/loader.py`](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/data/loader.py) (Data loader), [`src/train.py`](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/train.py) (Training workflow)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Corpus ingestion"]
  node_fetcher["Paper fetcher<br/>[arxiv_fetcher.py]"]
  node_tfidf["TF-IDF keywords<br/>[tfidf.cpp]"]
  node_tfidf_bindings["TF-IDF bindings<br/>[bindings.cpp]"]
  node_normalizer["Text normalizer<br/>[normalizer.cpp]"]
  node_normalizer_bindings["Normalizer bindings<br/>[bindings.cpp]"]
  node_loader["Data loader<br/>[loader.py]"]
end

subgraph group_learning["Representation learning"]
  node_trainer["Training workflow<br/>[train.py]"]
  node_pair_dataset["Triplet dataset<br/>[dataset.py]"]
  node_loss["Triplet loss<br/>[loss.py]"]
  node_encoder["Paper encoder<br/>[encoder.py]"]
end

subgraph group_serving["RAG serving"]
  node_cli["Interactive CLI<br/>[main.py]"]
  node_engine["RAG engine<br/>[engine.py]"]
end

subgraph group_storage["Vector storage"]
  node_database[("PostgreSQL pgvector")]
  node_schema["Database schema<br/>[schema.sql]"]
  node_connection["Connection pool<br/>[connection.py]"]
end

node_researcher(("Researcher"))
node_huggingface{{"HuggingFace datasets"}}
node_arxiv{{"ArXiv papers"}}
node_gemini{{"Google Gemini API"}}
node_answer["Cited answer"]

node_huggingface -->|"provides papers"| node_fetcher
node_arxiv -->|"paper source"| node_fetcher
node_fetcher -->|"supplies papers"| node_loader
node_loader -->|"bulk loads"| node_database
node_tfidf_bindings -->|"exposes extractor"| node_tfidf
node_tfidf -->|"provides keywords"| node_loader
node_normalizer_bindings -->|"exposes cleaner"| node_normalizer
node_trainer -->|"reads corpus data"| node_loader
node_trainer -->|"builds training data"| node_pair_dataset
node_trainer -->|"trains projection"| node_encoder
node_trainer -->|"optimizes with"| node_loss
node_pair_dataset -->|"uses keyword signals"| node_tfidf
node_trainer -->|"reads embeddings"| node_database
node_schema -->|"defines tables"| node_database
node_cli -->|"initializes"| node_normalizer
node_cli -->|"loads weights"| node_encoder
node_cli -->|"initializes pool"| node_connection
node_connection -->|"connects"| node_database
node_cli -->|"submits query"| node_engine
node_engine -->|"cleans query"| node_normalizer
node_engine -->|"embeds query"| node_encoder
node_engine -->|"requests retrieval"| node_loader
node_loader -->|"queries vectors"| node_database
node_engine -->|"calls agent"| node_gemini
node_gemini -->|"dispatches search"| node_engine
node_engine -->|"returns grounded text"| node_answer
node_cli -->|"displays"| node_answer
node_researcher -->|"asks question"| node_cli

click node_fetcher "https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/data/arxiv_fetcher.py"
click node_tfidf "https://github.com/bazanov05/arxiv-rag-engine/blob/main/tfidf/tfidf.cpp"
click node_tfidf_bindings "https://github.com/bazanov05/arxiv-rag-engine/blob/main/tfidf/bindings.cpp"
click node_normalizer "https://github.com/bazanov05/arxiv-rag-engine/blob/main/normalizer/normalizer.cpp"
click node_normalizer_bindings "https://github.com/bazanov05/arxiv-rag-engine/blob/main/normalizer/bindings.cpp"
click node_loader "https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/data/loader.py"
click node_trainer "https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/train.py"
click node_pair_dataset "https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/data/dataset.py"
click node_loss "https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/model/loss.py"
click node_encoder "https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/model/encoder.py"
click node_schema "https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/db/schema.sql"
click node_connection "https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/db/connection.py"
click node_cli "https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/main.py"
click node_engine "https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/rag/engine.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_fetcher,node_tfidf,node_tfidf_bindings,node_normalizer,node_normalizer_bindings,node_loader toneBlue
class node_trainer,node_pair_dataset,node_loss,node_encoder toneAmber
class node_cli,node_engine,node_gemini toneMint
class node_database,node_schema,node_connection toneRose
class node_researcher,node_huggingface,node_arxiv,node_answer toneIndigo
```

## Components

### Corpus ingestion

- [Paper fetcher](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/data/arxiv_fetcher.py): `src/data/arxiv_fetcher.py`
- [TF-IDF keywords](https://github.com/bazanov05/arxiv-rag-engine/blob/main/tfidf/tfidf.cpp): `tfidf/tfidf.cpp`
- [TF-IDF bindings](https://github.com/bazanov05/arxiv-rag-engine/blob/main/tfidf/bindings.cpp): `tfidf/bindings.cpp`
- [Text normalizer](https://github.com/bazanov05/arxiv-rag-engine/blob/main/normalizer/normalizer.cpp): `normalizer/normalizer.cpp`
- [Normalizer bindings](https://github.com/bazanov05/arxiv-rag-engine/blob/main/normalizer/bindings.cpp): `normalizer/bindings.cpp`
- [Data loader](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/data/loader.py): `src/data/loader.py`

### Representation learning

- [Training workflow](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/train.py): `src/train.py`
- [Triplet dataset](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/data/dataset.py): `src/data/dataset.py`
- [Triplet loss](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/model/loss.py): `src/model/loss.py`
- [Paper encoder](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/model/encoder.py): `src/model/encoder.py`

### RAG serving

- [Interactive CLI](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/main.py): `src/main.py`
- [RAG engine](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/rag/engine.py): `src/rag/engine.py`

### Vector storage

- **PostgreSQL pgvector**
- [Database schema](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/db/schema.sql): `src/db/schema.sql`
- [Connection pool](https://github.com/bazanov05/arxiv-rag-engine/blob/main/src/db/connection.py): `src/db/connection.py`

### Other components

- **Researcher**
- **HuggingFace datasets**
- **ArXiv papers**
- **Google Gemini API**
- **Cited answer**

## Connections

- HuggingFace datasets → Paper fetcher: provides papers
- ArXiv papers → Paper fetcher: paper source
- Paper fetcher → Data loader: supplies papers
- Data loader → PostgreSQL pgvector: bulk loads
- TF-IDF bindings → TF-IDF keywords: exposes extractor
- TF-IDF keywords → Data loader: provides keywords
- Normalizer bindings → Text normalizer: exposes cleaner
- Training workflow → Data loader: reads corpus data
- Training workflow → Triplet dataset: builds training data
- Training workflow → Paper encoder: trains projection
- Training workflow → Triplet loss: optimizes with
- Triplet dataset → TF-IDF keywords: uses keyword signals
- Training workflow → PostgreSQL pgvector: reads embeddings
- Database schema → PostgreSQL pgvector: defines tables
- Interactive CLI → Text normalizer: initializes
- Interactive CLI → Paper encoder: loads weights
- Interactive CLI → Connection pool: initializes pool
- Connection pool → PostgreSQL pgvector: connects
- Interactive CLI → RAG engine: submits query
- RAG engine → Text normalizer: cleans query
- RAG engine → Paper encoder: embeds query
- RAG engine → Data loader: requests retrieval
- Data loader → PostgreSQL pgvector: queries vectors
- RAG engine → Google Gemini API: calls agent
- Google Gemini API → RAG engine: dispatches search
- RAG engine → Cited answer: returns grounded text
- Interactive CLI → Cited answer: displays
- Researcher → Interactive CLI: asks question