# gair-nlp/openresearcher architecture

> Architecture diagram of the GitHub repository gair-nlp/openresearcher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gair-nlp/openresearcher
- Repository: https://github.com/gair-nlp/openresearcher

## Overview

##### Project type and purpose
- OpenResearcher is a Retrieval-Augmented Generation (RAG) system specialized as a scientific research assistant.
- It ingests arXiv papers, indexes them in a vector store (Qdrant) and a metadata search engine (Elasticsearch), and serves them to a chat-based UI backed by an LLM.

##### File structure
- **Top-level scripts and config**:
- config.py – holds API endpoints/keys for LLM, Bing Search, Qdrant/Elasticsearch
- requirements.txt, ui_app.py – entrypoint for the Streamlit UI
- **Connector** (connector/…): offline data ingestion, turns HTML + metadata into embeddings and indexes
- **Data** (data/…): raw arXiv HTML papers and metadata snapshot
- **Tools** (tool/…): small wrappers for BM25, PDF, code execution, web browsing, math, Qdrant
- **LLM wrapper** (llm/chat_llm.py): abstracted call to any OpenAI-compatible LLM or VLLM endpoint
- **Async Retriever Servers** (utils/…): run Qdrant and Elasticsearch retrievers as async micro-services
- **Service** (service/…): business logic modules – query decomposition, routing, hybrid retrieval, summarization, citation extraction, self-critique
- **Scripts** (scripts/…): helper shell for Qdrant indexing
- **Example_history**, **evaluation_data**, **images**: examples, benchmarks, assets

##### Key README insights
- Describes a three-stage data flow: ingest & index (offline), spin up retriever services, run the RAG chat UI
- Dependencies: Qdrant (vector DB), Elasticsearch, LLM APIs (OpenAI/Deepseek/vllm), optional Bing Search

1. Data Ingestion Pipeline
- Raw arXiv HTML + metadata (data/)
- connector.html_parsing → Qdrant
- connector.meta_elastic → Elasticsearch
2. Storage & Indexes
- Qdrant Vector Store
- Elasticsearch Metadata Index
3. Retriever Micro-services
- Async Qdrant Retriever (utils/async_qdrant_retriever.py)
- Async Elasticsearch Retriever (utils/async_elasticsearch_retriever.py)
4. Orchestrator & Business Logic
- Query Router (service/query_router.py)
- Query Decomposition & Sub-queries (service/query_decomposition.py, sub_query_response.py)
- Hybrid Retriever (service/hybrid_retriever.py)
- Summarizer & Self-Critic (service/summerize.py, self_critic.py)
- Other services: citation extraction, field selector, date selector
5. Tooling Layer
- Tools invoked by orchestrator (tool/*.py): BM25, PDF parsing, web search, code execution, math, Qdrant tools
6. LLM Interface
- chat_llm.py → calls LLM API endpoint
7. UI Layer
- Streamlit app (ui_app.py)

b. Relationships & data flow
- Offline: Data → Connector → Qdrant & Elasticsearch
- Startup: Launch retriever servers → Idle waiting
- User Query (UI) → Query Router → (Async retrievers + Tools + Bing Search) → Retrieved chunks
→ chat_llm for generation → Summarizer/Self-Critic → UI response

c. Architectural patterns & principles
- Micro-services for retrieval (separate async servers)
- Modular “service” layer encapsulating each RAG step
- Pluggable LLM backends via unified interface
- Clear separation: offline indexing vs. online serving vs. UI

d. Key technologies & frameworks
- Python, Streamlit (UI)
- Qdrant (vector store), Elasticsearch (text search)
- AsyncIO for retriever servers
- OpenAI-compatible API or VLLM for LLM
- Docker for Qdrant & Elasticsearch
- Shell scripts for orchestration

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Offline Indexing Lane
    subgraph "Offline Indexing"
        direction TB
        CFG["config.py"]:::service
        RAW["Raw arXiv Data (data/)"]:::service
        SP["save_qdrant_indexing.sh"]:::service
        HP["HTML Parsing Connector (connector/html_parsing.py)"]:::service
        ME["Metadata → Elasticsearch Connector (connector/meta_elastic.py)"]:::service
        QH["Qdrant Vector Store"]:::storage
        ESIdx["Elasticsearch Metadata Index"]:::storage

        RAW --> HP
        RAW --> ME
        HP --> QH
        ME --> ESIdx
        SP --> HP
        SP --> ME
    end

    %% Online Serving Lane
    subgraph "Online Serving"
        direction TB
        UI["Streamlit UI (ui_app.py)"]:::service
        ORC["Orchestrator & Business Logic"]:::service

        subgraph "Orchestrator & Business Logic"
            direction TB
            QR["Query Router (service/query_router.py)"]:::service
            QD["Query Decomposition (service/query_decomposition.py)"]:::service
            SQR["Sub-query Aggregation (service/sub_query_response.py)"]:::service
            HR["Hybrid Retriever (service/hybrid_retriever.py)"]:::service
            SUM["Summarization (service/summerize.py)"]:::service
            SC["Self-Critic (service/self_critic.py)"]:::service
            CE["Citation Extraction (service/add_citation.py)"]:::service
            FS["Field Selector (service/field_selector.py)"]:::service
            DS["Date Selector (service/date_selector.py)"]:::service
            ESR["ElasticSearch Retriever (service/elastic_search_retriever.py)"]:::service
            QRD["Qdrant Retriever (service/qdrant_retriever.py)"]:::service
            QI["Query Internet (service/query_internet.py)"]:::service
            QU["Query Understanding (service/query_understanding.py)"]:::service
            NA["Node Arrangement (service/nodes_arrangement.py)"]:::service
            MEA["Metadata Extraction (service/extractor.py)"]:::service
        end

        subgraph "Retriever Microservices"
            direction TB
            AQR["Async Qdrant Retriever (utils/async_qdrant_retriever.py)"]:::service
            AER["Async Elasticsearch Retriever (utils/async_elasticsearch_retriever.py)"]:::service
            AEE["Async Embedding Engine (utils/async_embedding_engine.py)"]:::service
        end

        subgraph "Tooling Layer"
            direction TB
            BM25["BM25 Tool (tool/bm25_tool.py)"]:::tools
            PDFT["PDF Parsing Tool (tool/chat_pdf_tool.py)"]:::tools
            CODE["Code Execution Tool (tool/code_tool.py)"]:::tools
            WEB["Web Browsing Tool (tool/internet_tool.py)"]:::tools
            MATH["Math Tool (tool/math_tool.py)"]:::tools
            QTOOL["Qdrant Utility Tool (tool/qdrant_tool.py)"]:::tools
        end

        LLMInt["LLM Interface (llm/chat_llm.py)"]:::service
        SS["Sentence Splitter (utils/sentence_splitter.py)"]:::service
        LLMAPI["LLM API Endpoint"]:::external
        BING["Bing Search API"]:::external

        UI --> QR
        QR -->|routes to| QD
        QD --> SQR
        SQR --> HR
        HR --> QRD
        HR --> ESR
        QR --> AQR
        QR --> AER
        QR --> AEE
        QR --> QI
        QR --> BM25
        QR --> PDFT
        QR --> CODE
        QR --> WEB
        QR --> MATH
        QR --> QTOOL
        QR --> SS

        ESR --> AER
        QRD --> AQR
        AEE --> QH
        QI --> BING

        HR --> LLMInt
        LLMInt --> LLMAPI
        LLMInt --> SUM
        SUM --> SC
        SC --> UI
    end

    %% Storage Services
    QH["Qdrant Vector Store"]:::storage
    ESIdx["Elasticsearch Metadata Index"]:::storage

    QR --> ESIdx
    QR --> QH

    CFG --> QR
    CFG --> UI
    CFG --> AQR
    CFG --> AER
    CFG --> AEE
    CFG --> LLMInt

    %% Click Events
    click CFG "https://github.com/gair-nlp/openresearcher/blob/main/config.py"
    click UI "https://github.com/gair-nlp/openresearcher/blob/main/ui_app.py"
    click RAW "https://github.com/gair-nlp/openresearcher/tree/main/data/"
    click HP "https://github.com/gair-nlp/openresearcher/blob/main/connector/html_parsing.py"
    click ME "https://github.com/gair-nlp/openresearcher/blob/main/connector/meta_elastic.py"
    click AQR "https://github.com/gair-nlp/openresearcher/blob/main/utils/async_qdrant_retriever.py"
    click AER "https://github.com/gair-nlp/openresearcher/blob/main/utils/async_elasticsearch_retriever.py"
    click AEE "https://github.com/gair-nlp/openresearcher/blob/main/utils/async_embedding_engine.py"
    click QR "https://github.com/gair-nlp/openresearcher/blob/main/service/query_router.py"
    click QD "https://github.com/gair-nlp/openresearcher/blob/main/service/query_decomposition.py"
    click SQR "https://github.com/gair-nlp/openresearcher/blob/main/service/sub_query_response.py"
    click HR "https://github.com/gair-nlp/openresearcher/blob/main/service/hybrid_retriever.py"
    click SUM "https://github.com/gair-nlp/openresearcher/blob/main/service/summerize.py"
    click SC "https://github.com/gair-nlp/openresearcher/blob/main/service/self_critic.py"
    click CE "https://github.com/gair-nlp/openresearcher/blob/main/service/add_citation.py"
    click FS "https://github.com/gair-nlp/openresearcher/blob/main/service/field_selector.py"
    click DS "https://github.com/gair-nlp/openresearcher/blob/main/service/date_selector.py"
    click ESR "https://github.com/gair-nlp/openresearcher/blob/main/service/elastic_search_retriever.py"
    click QRD "https://github.com/gair-nlp/openresearcher/blob/main/service/qdrant_retriever.py"
    click QI "https://github.com/gair-nlp/openresearcher/blob/main/service/query_internet.py"
    click QU "https://github.com/gair-nlp/openresearcher/blob/main/service/query_understanding.py"
    click NA "https://github.com/gair-nlp/openresearcher/blob/main/service/nodes_arrangement.py"
    click MEA "https://github.com/gair-nlp/openresearcher/blob/main/service/extractor.py"
    click BM25 "https://github.com/gair-nlp/openresearcher/blob/main/tool/bm25_tool.py"
    click PDFT "https://github.com/gair-nlp/openresearcher/blob/main/tool/chat_pdf_tool.py"
    click CODE "https://github.com/gair-nlp/openresearcher/blob/main/tool/code_tool.py"
    click WEB "https://github.com/gair-nlp/openresearcher/blob/main/tool/internet_tool.py"
    click MATH "https://github.com/gair-nlp/openresearcher/blob/main/tool/math_tool.py"
    click QTOOL "https://github.com/gair-nlp/openresearcher/blob/main/tool/qdrant_tool.py"
    click LLMInt "https://github.com/gair-nlp/openresearcher/blob/main/llm/chat_llm.py"
    click SS "https://github.com/gair-nlp/openresearcher/blob/main/utils/sentence_splitter.py"
    click SP "https://github.com/gair-nlp/openresearcher/blob/main/scripts/save_qdrant_indexing.sh"

    %% Styles
    classDef storage fill:#ADE1F0,stroke:#333,stroke-width:1px
    classDef service fill:#A0D995,stroke:#333,stroke-width:1px
    classDef tools fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef external fill:#F9D27A,stroke:#333,stroke-width:1px
```
