# shalinvachheta017/ragbot-core architecture

> Architecture diagram of the GitHub repository shalinvachheta017/ragbot-core, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shalinvachheta017/ragbot-core
- Repository: https://github.com/shalinvachheta017/ragbot-core

## Overview

##### Project Type & Purpose
- This is a full-stack, offline Retrieval-Augmented Generation (RAG) system specialized for German public‐tender documents with multilingual support.
- It ingests raw documents (PDF, Excel, DOCX, scanned), cleans and chunks them, persists embeddings in Qdrant, and serves a Streamlit UI for querying via an LLM (Ollama).

##### Core Components
a. Ingestion & Preprocessing
- scripts/parse_excel.py, core/io.py, core/domain.py
- Cleaning & metadata extraction
- OCR‐ready support for scanned files
b. Chunking & Embeddings
- scripts/embed_chunks_qdrant.py (or direct_embedding_pipeline.py)
- GPU-accelerated embedding models (E5-small, BGE, etc.) via core/search.py
c. Vector Database
- Qdrant vector store managed via docker-compose.yml
- scripts/ensure_qdrant.py for readiness checks
d. Retrieval & Reranking
- Retriever (top-K search) in core/search.py
- Cross-encoder reranker (ms-marco, BGE) in core/qa.py
e. LLM Answer Generation
- core/qa.py uses Ollama backends (Qwen2.5, Llama, Mistral)
- Produces structured, cited, explainable answers
f. Streamlit UI
- ui/app_streamlit.py, ui/tender_bot_ui.py, ui/multilingual_tender_bot.py
- Sidebar controls (model selection, filters, top-K)

##### Relationships & Data Flow
1. User inputs query in Streamlit UI
2. UI calls Retriever → Qdrant
3. Retrieved embeddings pass to Reranker → order top candidates
4. Top candidates + query go to LLM via Ollama
5. LLM returns structured answer + source citations
6. UI renders the final response
7. (Offline only) Batch pipelines feed data into Qdrant

##### Architectural Patterns & Principles
- Pipeline design: clear stages (Parse → Chunk → Embed → Store → Retrieve → Rerank → Generate)
- Separation of concerns: “scripts” for batch processes, “core” for domain logic, “ui” for presentation
- Privacy-first: No external API calls; all services run locally

##### Key Technologies & Frameworks
- Python 3.x, LangChain pipelines
- Qdrant (Docker Compose orchestration)
- Embedding models (Hugging Face E5, BGE)
- Cross-encoders for reranking (ms-marco, BGE)
- Ollama LLM backend
- Streamlit for UI

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Raw Documents
    Doc["Raw Tender Documents\n(PDF, DOCX, Excel, Scanned)"]:::external

    %% Ingestion & Preprocessing
    subgraph "Ingestion & Preprocessing"
        direction TB
        PE["parse_excel.py"]:::ingestion
        ING["ingest.py"]:::ingestion
        UDP["unified_document_processor.py"]:::ingestion
        IO["core/io.py"]:::ingestion
        DOM["core/domain.py"]:::ingestion
    end

    %% Chunking & Embeddings
    subgraph "Chunking & Embeddings"
        direction TB
        EDP["direct_embedding_pipeline.py"]:::embedding
        EMB["embed.py"]:::embedding
        SRCH["core/search.py"]:::embedding
    end

    %% Vector Database
    subgraph "Vector Database (Local Docker)" 
        direction TB
        QR["Qdrant\n(cylinder icon)"]:::database
    end

    %% Retrieval & Reranking
    subgraph "Retrieval & Reranking"
        direction TB
        RET["Retriever\n(core/search.py)"]:::retrieval
        RR["Reranker\n(core/qa.py)"]:::retrieval
        SCS["search.py"]:::retrieval
    end

    %% LLM Answer Generation
    subgraph "LLM Answer Generation"
        direction TB
        LLM["Ollama Backend\n(core/qa.py)"]:::llm
    end

    %% Streamlit UI
    subgraph "Streamlit UI"
        direction TB
        UI1["app_streamlit.py"]:::ui
        UI2["tender_bot_ui.py"]:::ui
        UI3["multilingual_tender_bot.py"]:::ui
    end

    %% Data Flows
    Doc -->|raw docs| PE
    PE -->|cleaned documents| EDP
    ING -->|unified docs| EDP
    UDP -->|processed docs| EDP
    IO -->|I/O ops| EDP
    DOM -->|metadata| EDP

    EDP -->|chunks & embeddings| QR
    EMB -->|embeddings| QR
    SRCH -->|embed models| QR

    %% Batch Ingestion auxiliary
    PE -.->|batch ingestion| QR
    ING -.->|batch ingestion| QR
    UDP -.->|batch ingestion| QR

    %% Retrieval Flow
    UI3 -->|user query| RET
    UI2 -->|user query| RET
    UI1 -->|user query| RET

    RET -->|query DB| QR
    QR -->|top-K vectors| RET
    RET -->|top-K IDs| RR
    RR -->|reranked candidates| LLM
    LLM -->|answer + citations| UI1
    LLM -->|answer + citations| UI2
    LLM -->|answer + citations| UI3

    %% Click Events
    click PE "https://github.com/shalinvachheta017/ragbot-core/blob/main/scripts/parse_excel.py"
    click ING "https://github.com/shalinvachheta017/ragbot-core/blob/main/scripts/ingest.py"
    click UDP "https://github.com/shalinvachheta017/ragbot-core/blob/main/scripts/unified_document_processor.py"
    click IO "https://github.com/shalinvachheta017/ragbot-core/blob/main/core/io.py"
    click DOM "https://github.com/shalinvachheta017/ragbot-core/blob/main/core/domain.py"
    click EDP "https://github.com/shalinvachheta017/ragbot-core/blob/main/scripts/direct_embedding_pipeline.py"
    click EMB "https://github.com/shalinvachheta017/ragbot-core/blob/main/scripts/embed.py"
    click SRCH "https://github.com/shalinvachheta017/ragbot-core/blob/main/core/search.py"
    click QR "https://github.com/shalinvachheta017/ragbot-core/blob/main/docker-compose.yml"
    click QR "https://github.com/shalinvachheta017/ragbot-core/blob/main/rag_qdrant.py"
    click RET "https://github.com/shalinvachheta017/ragbot-core/blob/main/core/search.py"
    click RR "https://github.com/shalinvachheta017/ragbot-core/blob/main/core/qa.py"
    click SCS "https://github.com/shalinvachheta017/ragbot-core/blob/main/scripts/search.py"
    click LLM "https://github.com/shalinvachheta017/ragbot-core/blob/main/core/qa.py"
    click UI1 "https://github.com/shalinvachheta017/ragbot-core/blob/main/ui/app_streamlit.py"
    click UI2 "https://github.com/shalinvachheta017/ragbot-core/blob/main/ui/tender_bot_ui.py"
    click UI3 "https://github.com/shalinvachheta017/ragbot-core/blob/main/ui/multilingual_tender_bot.py"

    %% Styles
    classDef ingestion fill:#FFD699,stroke:#CC6600,color:#663300
    classDef embedding fill:#CCE5FF,stroke:#3399FF,color:#003366
    classDef database fill:#E5CCFF,stroke:#9933CC,color:#330033
    classDef retrieval fill:#CCFFCC,stroke:#33CC33,color:#003300
    classDef llm fill:#F2CCFF,stroke:#CC33CC,color:#330033
    classDef ui fill:#FFFFCC,stroke:#CCCC33,color:#333300
    classDef external fill:#FFFFFF,stroke:#000000,color:#000000,stroke-dasharray: 5 5
```
