# abdoulaye-sayouti/secure-offline-rag-system architecture

> Architecture diagram of the GitHub repository abdoulaye-sayouti/secure-offline-rag-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abdoulaye-sayouti/secure-offline-rag-system
- Repository: https://github.com/abdoulaye-sayouti/secure-offline-rag-system

## Overview

##### Project Type and Purpose
- This is a Retrieval-Augmented Generation (RAG) system implemented in Python.
- It ingests diverse document formats, preprocesses and chunks them, builds hybrid retrieval indices (dense and sparse), serves queries via a local LLM server, and generates answers.
- It includes CLI tools, a Jupyter notebook, and a Streamlit web UI.

##### Main Architectural Components
a. Data Ingestion & Preprocessing
- Source folders (data/raw) holding PDFs, text, CSVs, URLs
- `src/data/loaders/` for file conversion and web scraping
- `src/data/data_ingestion.py` and `src/data/data_preprocessing.py`
- Chunking via `jupyter_and_code_spliter.py` and helpers
b. Caching Layer
- `src/cache/cache.py`
- Stores raw chunks and embeddings on disk (`cache/` dir)
c. Retrieval Layer
- Sparse retrieval: BM25 index (via `rank_bm25` in `bm25_retriever.py`)
- Dense retrieval: FAISS vector store (via `vector_retriever.py`)
- Hybrid retriever (`hybrid_retriever.py`) combining both
d. Reranking Model
- `src/models/reranker.py` wraps a BGE reranker on top-k results
e. LLM Serving
- External LLM server (llamafile on port 8081)
- `src/response/response_generator.py` formats prompt + context
f. Client Interfaces
- CLI: `process_single_query.py`, `generate_submission_csv.py`, `initialize_rag.py`
- Jupyter notebook: `inference_notebook.ipynb`
- Streamlit app: `streamlit_app.py`

##### Relationships and Data Flows
- Ingestion → Preprocessing → Chunking → Cache
(raw files → text chunks → cached on disk)
- Cache → Indexing → Retrieval Indices
(chunks → embeddings → FAISS; chunks → TF–IDF → BM25)
- Query → Hybrid Retriever → Top-K Chunks → Reranker
- Reranked Chunks + Query → LLM Server → Generated Response
- Response → CLI / Notebook / Web UI → User
- Submission generator → reads test.csv → uses same pipeline → writes CSV

##### Architectural Patterns & Design Principles
- Modular pipeline: separate data, cache, retrieval, models, response
- Dual encoding for hybrid search (sparse + dense)
- Disk-based caching to minimize recomputation
- Configurable via YAML files (`config/init_config.yaml`, `config/process_config.yaml`)
- Thread-safe and batch processing for performance
- Clear separation between offline initialization (cache build) and online inference

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources Layer
    subgraph "1. Data Sources"
        direction TB
        DataRaw["data/raw/"]:::dataSource
        PDF["PDF"]:::dataSource
        CSV["CSV"]:::dataSource
        URL["URLs"]:::dataSource
    end

    %% Preprocessing & Caching Layer
    subgraph "2. Preprocessing & Caching"
        direction TB
        Ingestion["Data Ingestion"]:::process
        Preprocessing["Data Preprocessing"]:::process
        Chunking["Chunking"]:::process
        subgraph "Loaders"
            direction TB
            Converter["File Converter"]:::process
            Loader["File Loader"]:::process
            Scraper["Web Scraper"]:::process
            BaseLoader["Base Loader"]:::process
        end
        CacheStore["Cache (./cache)"]:::cache
    end

    %% Indexing & Retrieval Layer
    subgraph "3. Indexing & Retrieval"
        direction TB
        BM25Index["BM25 Index"]:::index
        FAISSIndex["FAISS Vector Store"]:::index
        Hybrid["Hybrid Retriever"]:::process
    end

    %% Reranking & Generation Layer
    subgraph "4. Reranking & Generation"
        direction TB
        Reranker["Reranker"]:::model
        LLMServer["LLM Server\n(port 8081)"]:::model
        ResponseGen["Response Generator"]:::model
    end

    %% Client Interfaces Layer
    subgraph "5. Client Interfaces"
        direction TB
        CLI["CLI Tools"]:::client
        Notebook["Jupyter Notebook"]:::client
        Streamlit["Streamlit App"]:::client
        Submission["Submission CSV Generator"]:::client
    end

    %% Config
    InitConfig["init_config.yaml"]:::config
    ProcConfig["process_config.yaml"]:::config

    %% Connections
    PDF --> DataRaw
    CSV --> DataRaw
    URL --> DataRaw

    DataRaw --> Ingestion
    Ingestion --> Preprocessing
    Preprocessing --> Chunking
    Chunking --> CacheStore
    CacheStore --> Preprocessing

    CacheStore --> BM25Index
    CacheStore --> FAISSIndex
    BM25Index --> Hybrid
    FAISSIndex --> Hybrid

    Hybrid --> Reranker
    Reranker --> LLMServer
    LLMServer --> ResponseGen

    ResponseGen --> CLI
    ResponseGen --> Notebook
    ResponseGen --> Streamlit
    ResponseGen --> Submission

    InitConfig --> Ingestion
    ProcConfig --> Hybrid

    %% Click Events
    click DataRaw "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/tree/main/data/raw/"
    click Ingestion "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/src/data/data_ingestion.py"
    click Preprocessing "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/src/data/data_preprocessing.py"
    click Chunking "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/src/data/jupyter_and_code_spliter.py"
    click Converter "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/src/data/loaders/file_converter.py"
    click Loader "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/src/data/loaders/file_loader.py"
    click Scraper "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/src/data/loaders/web_scraper.py"
    click BaseLoader "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/src/data/loaders/base.py"
    click CacheStore "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/src/cache/cache.py"
    click BM25Index "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/src/retrieval/bm25_retriever.py"
    click FAISSIndex "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/src/retrieval/vector_retriever.py"
    click Hybrid "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/src/retrieval/hybrid_retriever.py"
    click Reranker "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/src/models/reranker.py"
    click ResponseGen "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/src/response/response_generator.py"
    click CLI "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/initialize_rag.py"
    click CLI "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/process_single_query.py"
    click CLI "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/generate_submission_csv.py"
    click CLI "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/init_cache_data.py"
    click Notebook "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/inference_notebook.ipynb"
    click Streamlit "https://github.com/abdoulaye-sayouti/secure-offline-rag-system/blob/main/streamlit_app.py"

    %% Styles
    classDef dataSource fill:#cbe5ff,stroke:#2c7bb6,color:#084594;
    classDef process fill:#ffd8b1,stroke:#e6550d,color:#a63603,stroke-width:2px,shape:roundrect;
    classDef cache fill:#fe9929,stroke:#cc4c02,color:#662506,shape:cylinder;
    classDef index fill:#c7e9c0,stroke:#31a354,color:#006d2c,shape:cylinder;
    classDef model fill:#ddc1eb,stroke:#6a51a3,color:#3f007d,shape:hexagon;
    classDef client fill:#f0f0f0,stroke:#636363,color:#252525;
    classDef config fill:#ffffcc,stroke:#b35900,color:#8c2d04;
    class IniCfg,ProcCfg config
```
