choeyunbeom/arxiv_rag_system

How choeyunbeom/arxiv_rag_system works

This is an arXiv-paper question-answering product: users submit a question through the Streamlit interface or API, retrieve and rerank relevant paper chunks, then receive a grounded answer with citations and latency information. A separate ingestion workflow collects papers, parses PDFs, chunks text, and indexes it for retrieval. The repository also includes benchmark evaluation, synthetic Q&A generation, LoRA fine-tuning, and embedding-space visualization. Relationships for sampled runtime code are based on excerpts and the SOURCE INDEX; ingestion and some UI capabilities are documented in the README, while their implementation details were not sampled.

Main entry points: rag_chain.py RAG orchestrator · hybrid_retriever.py Hybrid retrieval · evaluate.py RAG evaluation · main.py FastAPI app

Components 22

Paper ingestion

arXiv crawler src/ingestion/arxiv_crawler.py
PDF parser src/ingestion/pdf_parser.py
Text chunker src/api/core/chunker.py
Corpus indexer src/ingestion/indexer.py

Query service

FastAPI app src/api/main.py
Query endpoints src/api/routers/query.py
Health endpoint src/api/routers/health.py
API schemas src/api/models/schemas.py
Runtime settings src/api/core/config.py

Retrieval and generation

RAG orchestrator src/api/core/rag_chain.py
Hybrid retrieval src/api/core/hybrid_retriever.py
Answer prompts src/api/core/prompts.py
LLM client src/api/core/llm_client.py

ML experiments

RAG evaluation src/evaluation/evaluate.py
Benchmark dataset src/evaluation/eval_dataset.py
Synthetic Q&A src/finetuning/generate_qa_dataset.py
LoRA training src/finetuning/finetune_lora.ipynb

User experience

Streamlit UI ui/app.py
Embedding visualization scripts/precompute_umap.py

Other components

Question asker
Ollama service
ChromaDB

Connections 22

Question asker → Streamlit UI: asks through README.md
Streamlit UI → FastAPI app: submits query README.md
FastAPI app → Query endpoints: mounts router src/api/main.py
FastAPI app → Health endpoint: mounts router src/api/main.py
Query endpoints → RAG orchestrator: runs query src/api/routers/query.py
Query endpoints → API schemas: validates and returns src/api/routers/query.py
RAG orchestrator → Hybrid retrieval: retrieves chunks src/api/core/rag_chain.py
RAG orchestrator → LLM client: generates answer src/api/core/rag_chain.py
RAG orchestrator → Answer prompts: uses templates src/api/core/rag_chain.py
Hybrid retrieval → ChromaDB: searches vectors src/api/core/hybrid_retriever.py
Hybrid retrieval → Ollama service: embeds query src/api/core/hybrid_retriever.py
LLM client → Ollama service: generates text src/api/core/llm_client.py
arXiv crawler → PDF parser: provides PDFs README.md
PDF parser → Text chunker: produces parsed text README.md
Text chunker → Corpus indexer: provides chunks README.md
Corpus indexer → ChromaDB: indexes corpus README.md
RAG evaluation → Benchmark dataset: benchmarks with src/evaluation/evaluate.py
RAG evaluation → Hybrid retrieval: measures retrieval src/evaluation/evaluate.py
RAG evaluation → RAG orchestrator: measures answers src/evaluation/evaluate.py
Synthetic Q&A → Ollama service: generates examples src/finetuning/generate_qa_dataset.py
Synthetic Q&A → LoRA training: prepares training data README.md
Streamlit UI → Embedding visualization: uses precomputed map README.md

Drawn by GitDiagram on November 14, 2026 from the repository's file tree, README and source files. Read as Markdown ·