# yash1802/on-device-research-assistant-powered-by-rag-and-live-web-search architecture

> Architecture diagram of the GitHub repository yash1802/on-device-research-assistant-powered-by-rag-and-live-web-search, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yash1802/on-device-research-assistant-powered-by-rag-and-live-web-search
- Repository: https://github.com/yash1802/on-device-research-assistant-powered-by-rag-and-live-web-search

## Overview

##### Project type and purpose
- This is a full-stack RAG (Retrieval-Augmented Generation) prototype with both a user interface (Streamlit) and multiple backend pipelines for Web scraping, PDF ingestion, retrieval, embedding storage, and optional LLM answer synthesis via Google Gemma.
- Purpose: allow end users to query the web and/or uploaded PDFs, retrieve relevant text chunks, embed them, and optionally feed a language model to produce grounded answers.

##### File structure
- Top-level UI:
- Streamlit_Driver.py – orchestrates UI, mode selection, invokes pipelines
- Backend pipelines:
- Web_RAG_Model.py – web-only RAG pipeline
- Pdf_RAG_Model.py – PDF-only RAG pipeline
- PDF_Web_RAG_Model.py – combined Web+PDF retrieval
- LLM_Pdf_Web_RAG.py – combined RAG retrieval + LLM answer synthesis
- LLM_Module.py – direct LLM querying (no RAG)
- Shared config:
- ImportsForModel.py – centralizes imports, device detection, quantization defaults
- Data storage (created at runtime):
- EmbeddingStorage/ – serialized embeddings (.pkl)
- cache/ – metadata about scraped URLs
- scraped_texts/ – raw text of scraped pages
- requirements.txt – lists external libraries: streamlit, sentence-transformers, PyMuPDF, faiss, bitsandbytes, etc.

##### README for additional insights
- Detailed description of web pipeline (googlesearch-python, BeautifulSoup, boilerplate removal, language filter, chunking)
- PDF pipeline (PyMuPDF, chapter detection, overlapping chunking)
- Retrieval algorithms: dot-product, cosine, Euclidean, FAISS, BM25+embeddings, cross-encoder reranking
- LLM integration: Google Gemma 2B/7B via HF Transformers, FlashAttention2, optional 4-bit quantization
- UI modes: five search modes selecting different pipelines or LLM paths

a. Main components
- User Interface (Streamlit Driver)
- Mode Router / Controller
- Web Scraper & Processor
- PDF Ingestion & Processor
- Chunking Module
- Embedding Module (all-mpnet-base-v2)
- Retrieval Engine(s):
- Similarity Search (dot, cos, Euclid)
- FAISS Index
- BM25 + Dense Hybrid
- Cross-Encoder Reranker
- LLM Module (Google Gemma via Transformers)
- Storage Layer (file system): EmbeddingStorage, cache, scraped_texts
- External Services: Google Search API, Websites, Hugging Face Hub
b. Relationships & data flow
1. UI → Controller: captures query, mode, PDF upload
2. Controller → appropriate pipeline module
3. Pipeline →
- If Web: call Web Scraper → clean HTML → chunk
- If PDF: call PDF Processor → extract text → chunk
- If combined: do both streams in parallel or sequence
4. Chunks → Embedding Module → persist/fetch embeddings in EmbeddingStorage
5. Embeddings → Retrieval Engine(s) → top k chunks
6. If LLM mode: top chunks → LLM Module → generate answer
7. Controller → UI: return chunks and/or LLM answer
c. Architectural patterns & principles
- Layered architecture: UI layer, application/controller layer, processing layer, storage layer
- Pipeline pattern: each data source → clean → chunk → embed → retrieve → (optional) generate
- Separation of concerns: distinct modules for Web, PDF, embeddings, retrieval, LLM
- Config-driven behavior: ImportsForModel centralizes hardware/quantization logic
d. Key technologies & libraries
- Streamlit for UI
- googlesearch-python, readability-lxml, BeautifulSoup4 for web
- PyMuPDF (fitz) for PDF
- spaCy or NLTK for sentence segmentation
- sentence-transformers (“all-mpnet-base-v2”) for embeddings
- faiss-cpu (or faiss-gpu) for ANN search
- rank-bm25 for lexical retrieval
- cross-encoder/ms-marco-MiniLM-L-6-v2 for reranking
- Hugging Face Transformers + bitsandbytes for Gemma LLM

##### Detail without over-modularizing
- That ImportsForModel underpins all model and hardware configuration steps.
- Runtime folder creation (cache/, scraped_texts/) as part of the storage component.
- Fallback paths (e.g., 4-bit quantization disabled → float16 path).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% UI Layer
    UI_Controller["Streamlit Driver\n(UI / Mode Router)"]:::frontend

    %% Processing Layer
    subgraph "Processing Layer"
        Imports["ImportsForModel\n(Config)"]:::processing
        WebRAG["Web Scraper & RAG Pipeline"]:::processing
        PdfRAG["PDF Ingestion & RAG Pipeline"]:::processing
        CombinedRAG["Combined Web+PDF Retrieval"]:::processing
        LLMRAGSynth["RAG Retrieval + LLM Answer Synthesis"]:::processing
        DirectLLM["Direct LLM Querying Module"]:::processing
        Chunker["Chunking Module"]:::processing
        Embed["Embedding Module\n(all-mpnet-base-v2)"]:::processing
        Retrieval["Retrieval Engines\n(dot, cos, Euclid, FAISS, BM25, Reranker)"]:::processing
        LLMModule["LLM Module\n(Google Gemma)"]:::processing
    end

    %% Storage Layer
    subgraph "Storage Layer"
        ES((EmbeddingStorage)):::storage
        Cache((cache)):::storage
        Scraped((scraped_texts)):::storage
    end

    %% External Services
    subgraph "External Services"
        GoogleSearch["Google Search API"]:::external
        Websites["Websites"]:::external
        HFHub["Hugging Face Hub"]:::external
    end

    %% Data Flows
    UI_Controller -->|select mode, query, upload| WebRAG
    UI_Controller -->|select mode, query, upload| PdfRAG
    UI_Controller -->|select mode, query, upload| CombinedRAG
    UI_Controller -->|select mode, query, upload| LLMRAGSynth
    UI_Controller -->|select mode, query| DirectLLM

    WebRAG -->|HTTP, BeautifulSoup| GoogleSearch
    WebRAG -->|scrape pages| Websites
    WebRAG -->|text→chunk| Chunker

    PdfRAG -->|extract text| Chunker

    CombinedRAG -->|web+pdf| WebRAG
    CombinedRAG -->|web+pdf| PdfRAG

    LLMRAGSynth -->|retrieve chunks| Retrieval
    LLMRAGSynth -->|generate answer| LLMModule

    DirectLLM -->|query LLM| LLMModule

    Chunker -->|chunks| Embed
    Embed -->|.pkl| ES
    Embed -->|load/persist| ES

    ES -->|vectors| Retrieval
    Cache -->|metadata| Retrieval
    Scraped -->|raw text| Retrieval

    Retrieval -->|top-k chunks| LLMModule
    LLMModule -->|answer| UI_Controller
    Retrieval -->|chunks| UI_Controller

    %% Config dependency
    UI_Controller --> Imports
    WebRAG --> Imports
    PdfRAG --> Imports
    CombinedRAG --> Imports
    LLMRAGSynth --> Imports
    DirectLLM --> Imports

    %% Click Events
    click UI_Controller "https://github.com/yash1802/on-device-research-assistant-powered-by-rag-and-live-web-search/blob/main/Streamlit_Driver.py"
    click Imports "https://github.com/yash1802/on-device-research-assistant-powered-by-rag-and-live-web-search/blob/main/ImportsForModel.py"
    click WebRAG "https://github.com/yash1802/on-device-research-assistant-powered-by-rag-and-live-web-search/blob/main/Web_RAG_Model.py"
    click PdfRAG "https://github.com/yash1802/on-device-research-assistant-powered-by-rag-and-live-web-search/blob/main/Pdf_RAG_Model.py"
    click CombinedRAG "https://github.com/yash1802/on-device-research-assistant-powered-by-rag-and-live-web-search/blob/main/PDF_Web_RAG_Model.py"
    click LLMRAGSynth "https://github.com/yash1802/on-device-research-assistant-powered-by-rag-and-live-web-search/blob/main/LLM_Pdf_Web_RAG.py"
    click DirectLLM "https://github.com/yash1802/on-device-research-assistant-powered-by-rag-and-live-web-search/blob/main/LLM_Module.py"

    %% Styles
    classDef frontend fill:#ADD8E6,stroke:#333,stroke-width:1px;
    classDef processing fill:#90EE90,stroke:#333,stroke-width:1px;
    classDef storage fill:#D3D3D3,stroke:#333,stroke-width:1px;
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px;
```
