How muhammadahmedyaqoob/saudi-food-and-drug-authority-rag-system works
Project Type and Purpose
• This is a Python-based Retrieval-Augmented Generation (RAG) system with a Streamlit UI.
• Its goal is to let users (in Arabic or English) query a heterogeneous knowledge graph built from a corpus of PDF documents (Saudi Food & Drug Authority specifications), retrieve relevant passages via combined vector & graph heuristics, and generate concise, citation-free answers.
File Structure
• Top level:
• data/ – raw artifacts: FAISS index, pickled graph mappings, knowledge graphs
• rags/ – source PDF documents
• src/ – core modules: loader, indexer, retriever, generator, config, main entrypoint, model list
• streamlit_app.py – front-end application
• requirements.txt – dependencies (FAISS, spaCy, LLM client, Streamlit)
• src/ components indicate a classic pipeline:
• loader.py reads & preprocesses PDFs into text & entities
• graph_indexer.py builds/updates the heterogeneous knowledge graph
• retriever.py performs combined semantic + graph-heuristic retrieval against FAISS index
• generator.py stitches retrieved docs into prompts and calls LLM(s)
• list_models.py enumerates available LLM models
• config.py centralizes settings (paths, model names, language options)
• main.py orchestrates CLI runs (batch indexing, retrieval)
• streamlit_app.py provides the interactive UI with language switcher and graph visualization.
README for Additional Insights
• Bilingual support via spaCy models en_core_web_sm & xx_ent_wiki_sm
• FAISS directory holds prebuilt semantic.index and doc_mapping.pkl
• Knowledge graphs serialized as .pkl (bakery_graph, saudia_drugs_graph, sfda_knowledge_graph)
• System merges miniRAG–style graph retrieval with coRAG–style generation
a. Main Components
• User Interface: Streamlit front end
• Document Repository: PDFs under rags/
• Loader & Preprocessor: Text extraction + entity recognition (spaCy)
• Graph Indexer: Builds heterogeneous knowledge graph (NetworkX or similar)
• Vector Store: FAISS index for semantic search
• Retriever: Hybrid semantic + graph-based retrieval logic
• Generator: Prompts an LLM for answer generation (OpenAI, HuggingFace)
• Configuration Module: Central configurables (paths, model names)
• Model Directory: list_models component
• Knowledge Graph Storage: Serialized .pkl graph files
b. Relationships & Data Flow
1. User → Streamlit UI → Query (text + language choice)
4. UI displays answer and optional graph visualization
5. Offline/batch: loader → graph_indexer → updates FAISS & graph storage
c. Architectural Patterns & Principles
• Pipeline pattern: loader → indexer → retriever → generator
• Separation of concerns: distinct modules for data ingestion, indexing, retrieval, and generation
• Hybrid retrieval: semantic vectors + explicit graph heuristics for cross-document connections
• Pluggable LLM backends via list_models.py and config.py
d. Technologies, Frameworks, Libraries
• Python 3.10
• Streamlit (UI)
• spaCy (NER & multilingual tokenization)
• FAISS (vector database)
• NetworkX or similar for graph construction (implied)
• OpenAI or HuggingFace LLM client
• pickle for serialization
Detailed Capture of Architectural Elements
• Under “Generator,” note prompt assembly logic and citation-free policy.
• In the UI block, add a toggle for Arabic/English and a panel for graph visualization.