How abdoulaye-sayouti/secure-offline-rag-system works
Project Type and Purpose
• This is a Retrieval-Augmented Generation (RAG) system implemented in Python.
• It ingests diverse document formats, preprocesses and chunks them, builds hybrid retrieval indices (dense and sparse), serves queries via a local LLM server, and generates answers.
• It includes CLI tools, a Jupyter notebook, and a Streamlit web UI.
Main Architectural Components
a. Data Ingestion & Preprocessing
• Source folders (data/raw) holding PDFs, text, CSVs, URLs
• src/data/loaders/ for file conversion and web scraping
• src/data/data_ingestion.py and src/data/data_preprocessing.py
• Chunking via jupyter_and_code_spliter.py and helpers
b. Caching Layer
• src/cache/cache.py
• Stores raw chunks and embeddings on disk (cache/ dir)
c. Retrieval Layer
• Sparse retrieval: BM25 index (via rank_bm25 in bm25_retriever.py)
• Dense retrieval: FAISS vector store (via vector_retriever.py)
• Hybrid retriever (hybrid_retriever.py) combining both
d. Reranking Model
• src/models/reranker.py wraps a BGE reranker on top-k results
e. LLM Serving
• External LLM server (llamafile on port 8081)
• src/response/response_generator.py formats prompt + context
f. Client Interfaces
• CLI: process_single_query.py, generate_submission_csv.py, initialize_rag.py
• Jupyter notebook: inference_notebook.ipynb
• Streamlit app: streamlit_app.py
Relationships and Data Flows
• Ingestion → Preprocessing → Chunking → Cache
(raw files → text chunks → cached on disk)
• Cache → Indexing → Retrieval Indices
(chunks → embeddings → FAISS; chunks → TF–IDF → BM25)
• Query → Hybrid Retriever → Top-K Chunks → Reranker
• Reranked Chunks + Query → LLM Server → Generated Response
• Response → CLI / Notebook / Web UI → User
• Submission generator → reads test.csv → uses same pipeline → writes CSV
Architectural Patterns & Design Principles
• Modular pipeline: separate data, cache, retrieval, models, response
• Dual encoding for hybrid search (sparse + dense)
• Disk-based caching to minimize recomputation
• Configurable via YAML files (config/init_config.yaml, config/process_config.yaml)
• Thread-safe and batch processing for performance
• Clear separation between offline initialization (cache build) and online inference