How gair-nlp/openresearcher works
Project type and purpose
• OpenResearcher is a Retrieval-Augmented Generation (RAG) system specialized as a scientific research assistant.
• It ingests arXiv papers, indexes them in a vector store (Qdrant) and a metadata search engine (Elasticsearch), and serves them to a chat-based UI backed by an LLM.
File structure
• Top-level scripts and config:
• config.py – holds API endpoints/keys for LLM, Bing Search, Qdrant/Elasticsearch
• requirements.txt, ui_app.py – entrypoint for the Streamlit UI
• Connector (connector/…): offline data ingestion, turns HTML + metadata into embeddings and indexes
• Data (data/…): raw arXiv HTML papers and metadata snapshot
• Tools (tool/…): small wrappers for BM25, PDF, code execution, web browsing, math, Qdrant
• LLM wrapper (llm/chat_llm.py): abstracted call to any OpenAI-compatible LLM or VLLM endpoint
• Async Retriever Servers (utils/…): run Qdrant and Elasticsearch retrievers as async micro-services
• Service (service/…): business logic modules – query decomposition, routing, hybrid retrieval, summarization, citation extraction, self-critique
• Scripts (scripts/…): helper shell for Qdrant indexing
• Example_history, evaluation_data, images: examples, benchmarks, assets
Key README insights
• Describes a three-stage data flow: ingest & index (offline), spin up retriever services, run the RAG chat UI
• Dependencies: Qdrant (vector DB), Elasticsearch, LLM APIs (OpenAI/Deepseek/vllm), optional Bing Search
1. Data Ingestion Pipeline
• Raw arXiv HTML + metadata (data/)
• connector.html_parsing → Qdrant
• connector.meta_elastic → Elasticsearch
2. Storage & Indexes
• Qdrant Vector Store
• Elasticsearch Metadata Index
3. Retriever Micro-services
• Async Qdrant Retriever (utils/async_qdrant_retriever.py)
• Async Elasticsearch Retriever (utils/async_elasticsearch_retriever.py)
4. Orchestrator & Business Logic
• Query Router (service/query_router.py)
• Query Decomposition & Sub-queries (service/query_decomposition.py, sub_query_response.py)
• Hybrid Retriever (service/hybrid_retriever.py)
• Summarizer & Self-Critic (service/summerize.py, self_critic.py)
• Other services: citation extraction, field selector, date selector
5. Tooling Layer
• Tools invoked by orchestrator (tool/*.py): BM25, PDF parsing, web search, code execution, math, Qdrant tools
6. LLM Interface
• chat_llm.py → calls LLM API endpoint
7. UI Layer
• Streamlit app (ui_app.py)
b. Relationships & data flow
• Offline: Data → Connector → Qdrant & Elasticsearch
• Startup: Launch retriever servers → Idle waiting
• User Query (UI) → Query Router → (Async retrievers + Tools + Bing Search) → Retrieved chunks
→ chat_llm for generation → Summarizer/Self-Critic → UI response
c. Architectural patterns & principles
• Micro-services for retrieval (separate async servers)
• Modular “service” layer encapsulating each RAG step
• Pluggable LLM backends via unified interface
• Clear separation: offline indexing vs. online serving vs. UI
d. Key technologies & frameworks
• Python, Streamlit (UI)
• Qdrant (vector store), Elasticsearch (text search)
• AsyncIO for retriever servers
• OpenAI-compatible API or VLLM for LLM
• Docker for Qdrant & Elasticsearch
• Shell scripts for orchestration