How ginga1402/sanvad-hindi-rag-langchain-faiss-sarvam works
Project Type and Purpose
• Sanvad is a full-stack, document-centric Retrieval-Augmented Generation (RAG) system for Hindi-language Q&A.
• It combines a data-ingestion pipeline, a vector store, a LangChain-driven RAG engine with the Sarvam-m LLM, a Flask API, and an interactive Streamlit frontend.
File Structure
• Top-level scripts:
• ingest.py – PDF ingestion, text splitting, embedding, FAISS index creation
• hindi_rag.py – Core RAG pipeline (retriever + LLM chain)
• flask_app.py – REST API server exposing endpoints for queries
• streamlit_app.py – Web UI for interactive chat
• vector_store/ – persisted FAISS index files (index.faiss, index.pkl)
• __pycache__/ – Python bytecode (ignore)
• (Implicit) data/pdfs/ – source documents
Extract Key Architectural Insights from the README
• Uses LangChain modules: DocumentLoader, TextSplitter, Embeddings, FAISS vector store, LLMChain.
• FAISS for similarity search; Sarvam-m LLM via Hugging Face Transformers.
• Two access modes: programmatic REST API (Flask) and Streamlit UI.
Main Components
a. Data Ingestion Pipeline
• PDF Loader → Text Splitter → Embedding Generator → FAISS Vector Store
b. Vector Store (Persistent Layer)
• FAISS index files on disk
c. RAG Engine (hindi_rag.py)
• Retriever (FAISS) → Context Assembler → Sarvam-m LLM (via Transformers) → Post-processing
d. API Layer (flask_app.py)
• /ingest endpoint (optional trigger)
• /query endpoint → invokes RAG Engine → returns JSON response
e. Frontend (streamlit_app.py)
• User Input → HTTP POST to Flask API → Displays response
Component Relationships and Data Flows
• Ingestion Script → writes to Vector Store
• Flask App & Streamlit UI → call the RAG Engine
• RAG Engine → reads from Vector Store → calls LLM → returns answer
• Optionally, the ingestion endpoint in Flask can trigger vector store (if exposed)
Architectural Patterns and Design Principles
• Layered architecture: Data Layer (vector store), Business Logic Layer (RAG engine), Interface Layer (API + UI)
• Single-responsibility for each script
• Modular LangChain chains for easy extension
• File-based persistence for vector store, no external DB
Technologies and Frameworks to Annotate
• LangChain (chains, retriever)
• FAISS (vector similarity search)
• Sarvam-m LLM via Hugging Face Transformers
• Flask (REST API)
• Streamlit (UI)
• Tiktoken (token counting, optional)