# ginga1402/sanvad-hindi-rag-langchain-faiss-sarvam architecture

> Architecture diagram of the GitHub repository ginga1402/sanvad-hindi-rag-langchain-faiss-sarvam, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ginga1402/sanvad-hindi-rag-langchain-faiss-sarvam
- Repository: https://github.com/ginga1402/sanvad-hindi-rag-langchain-faiss-sarvam

## Overview

##### Project Type and Purpose
- Sanvad is a full-stack, document-centric Retrieval-Augmented Generation (RAG) system for Hindi-language Q&A.
- It combines a data-ingestion pipeline, a vector store, a LangChain-driven RAG engine with the Sarvam-m LLM, a Flask API, and an interactive Streamlit frontend.

##### File Structure
- Top-level scripts:
- ingest.py – PDF ingestion, text splitting, embedding, FAISS index creation
- hindi_rag.py – Core RAG pipeline (retriever + LLM chain)
- flask_app.py – REST API server exposing endpoints for queries
- streamlit_app.py – Web UI for interactive chat
- vector_store/ – persisted FAISS index files (index.faiss, index.pkl)
- __pycache__/ – Python bytecode (ignore)
- (Implicit) data/pdfs/ – source documents

##### Extract Key Architectural Insights from the README
- Uses LangChain modules: DocumentLoader, TextSplitter, Embeddings, FAISS vector store, LLMChain.
- FAISS for similarity search; Sarvam-m LLM via Hugging Face Transformers.
- Two access modes: programmatic REST API (Flask) and Streamlit UI.

##### Main Components
a. Data Ingestion Pipeline
- PDF Loader → Text Splitter → Embedding Generator → FAISS Vector Store
b. Vector Store (Persistent Layer)
- FAISS index files on disk
c. RAG Engine (hindi_rag.py)
- Retriever (FAISS) → Context Assembler → Sarvam-m LLM (via Transformers) → Post-processing
d. API Layer (flask_app.py)
- /ingest endpoint (optional trigger)
- /query endpoint → invokes RAG Engine → returns JSON response
e. Frontend (streamlit_app.py)
- User Input → HTTP POST to Flask API → Displays response

##### Component Relationships and Data Flows
- Ingestion Script → writes to Vector Store
- Flask App & Streamlit UI → call the RAG Engine
- RAG Engine → reads from Vector Store → calls LLM → returns answer
- Optionally, the ingestion endpoint in Flask can trigger vector store (if exposed)

##### Architectural Patterns and Design Principles
- Layered architecture: Data Layer (vector store), Business Logic Layer (RAG engine), Interface Layer (API + UI)
- Single-responsibility for each script
- Modular LangChain chains for easy extension
- File-based persistence for vector store, no external DB

##### Technologies and Frameworks to Annotate
- LangChain (chains, retriever)
- FAISS (vector similarity search)
- Sarvam-m LLM via Hugging Face Transformers
- Flask (REST API)
- Streamlit (UI)
- Tiktoken (token counting, optional)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        pdfs[/"data/pdfs/*.pdf"/]:::ingestion
        ingest["ingest.py"]:::ingestion
        loader["PDF Loader"]:::ingestion
        splitter["Text Splitter"]:::ingestion
        embed["Embedding Generator"]:::ingestion
        store["FAISS Vector Store"]:::ingestion
        store_dir(["vector_store/"]):::ingestion
    end

    pdfs --> loader --> splitter --> embed --> store --> store_dir

    %% Processing Layer
    subgraph "Processing (RAG Engine)"
        rag_script["hindi_rag.py"]:::processing
        retriever["Retriever (FAISS)"]:::processing
        assembler["Context Assembler"]:::processing
        llm["Sarvam-m LLM"]:::processing
        post["Post-Processing"]:::processing
    end

    store_dir --> retriever --> assembler --> llm --> post --> rag_script

    %% Interface Layer
    subgraph "Interface Layer"
        api["flask_app.py"]:::interface
        ui["streamlit_app.py"]:::interface
    end

    api -->|"POST /ingest"| ingest
    api -->|"POST /query"| rag_script
    ui -->|"HTTP POST to /query"| api
    ui -->|"imports RAG Engine"| rag_script

    %% Click Events
    click ingest "https://github.com/ginga1402/sanvad-hindi-rag-langchain-faiss-sarvam/blob/main/ingest.py"
    click store_dir "https://github.com/ginga1402/sanvad-hindi-rag-langchain-faiss-sarvam/tree/main/vector_store/"
    click rag_script "https://github.com/ginga1402/sanvad-hindi-rag-langchain-faiss-sarvam/blob/main/hindi_rag.py"
    click api "https://github.com/ginga1402/sanvad-hindi-rag-langchain-faiss-sarvam/blob/main/flask_app.py"
    click ui "https://github.com/ginga1402/sanvad-hindi-rag-langchain-faiss-sarvam/blob/main/streamlit_app.py"

    %% Styles
    classDef ingestion fill:#e0f7fa,stroke:#006064,color:#000;
    classDef processing fill:#e8f5e9,stroke:#2e7d32,color:#000;
    classDef interface fill:#fff3e0,stroke:#ef6c00,color:#000;
```
