# devarsh26/rag-app architecture

> Architecture diagram of the GitHub repository devarsh26/rag-app, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/devarsh26/rag-app
- Repository: https://github.com/devarsh26/rag-app

## Overview

##### Project Type & Purpose
- This is a full-stack application focused on AI-powered document querying, using a Retrieval-Augmented Generation (RAG) approach.
- The main goal is to reduce document query resolution time by automatically extracting answers from internal documents (PDFs, Word, JSON, XML).

##### Analyzing the Architectural Components
From the README, break down the overall workflow into these key components:
- Document Ingestion: Documents collected from SharePoint.
- Data Pipeline: Uses Kafka and PySpark to stream and process both structured and unstructured data.
- Data Storage: Files are stored in SQLite (and note there is mention of Databricks as well).
- Processing Pipeline:
– Chunking: Uses LangChain’s SemanticChunker to break text into smaller parts.
– Embedding: Using HuggingFace’s sentence-transformer (MiniLM) for generating embeddings.
– Vector Storage: The embedding vectors are stored in a FAISS index (and also SQLite is mentioned for storage).
- Query Engine:
– Retrieves relevant context from the vector database based on semantic similarity.
– A locally run Mistral LLM (accessed via Ollama) generates answers using the provided context combined with the query.
- User Interface:
– A Streamlit-based UI provides the front-end for employee interaction.

##### Putting It All Together
- Branch this into storage components: one path for raw files in SQLite/Databricks and a parallel path for processing that goes to “Chunking & Embedding” with LangChain and HuggingFace MiniLM.
- Continue by depicting the “Query Module” which queries the vector DB for relevant context.
- Direct the output of the query module into the “Mistral LLM (via Ollama)” block, showing the generation of answers.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Documentation Node
    Docs["System Documentation (README)"]:::docs
    Docs -.-> DocIngest["Document Ingestion (SharePoint)"]:::ingestion

    %% Subgraph: Ingestion and Data Pipeline
    subgraph "Ingestion & Pipeline"
        DocIngest["Document Ingestion (SharePoint)"]:::ingestion
        DataPipe["Data Pipeline (Kafka + PySpark)"]:::pipeline
        DocIngest -->|"transfers"| DataPipe
    end

    %% Subgraph: Data Storage & Processing
    subgraph "Data Storage & Processing"
        RawStore["Raw Document Storage (SQLite/Databricks)"]:::storage
        Process["Chunking & Embedding (LangChain & HuggingFace MiniLM)"]:::processing
        Embedding["Vector Storage (FAISS Vector DB)"]:::storage
        DataPipe -->|"sends raw data to"| RawStore
        DataPipe -->|"streams for processing"| Process
        Process -->|"produces embeddings"| Embedding
    end

    %% Subgraph: Query & Response
    subgraph "Query & Response"
        Query["Query Module (Context Retrieval)"]:::query
        LLM["Mistral LLM (via Ollama)"]:::llm
        UI["Streamlit Frontend"]:::ui
        Embedding -->|"provides context"| Query
        Query -->|"queries LLM"| LLM
        LLM -->|"delivers answer"| UI
    end

    %% Click Events
    click Docs "https://github.com/devarsh26/rag-app/blob/main/README.md"

    %% Styles
    classDef ingestion fill:#A2D2FF,stroke:#1D3557,stroke-width:2px;
    classDef pipeline fill:#BDE0FE,stroke:#1A759F,stroke-width:2px;
    classDef storage fill:#FFDAA9,stroke:#E63946,stroke-width:2px;
    classDef processing fill:#FFEFBA,stroke:#F4A261,stroke-width:2px;
    classDef query fill:#C9E4C5,stroke:#457B9D,stroke-width:2px;
    classDef llm fill:#FDD7E4,stroke:#E29578,stroke-width:2px;
    classDef ui fill:#D8E2DC,stroke:#95A3B3,stroke-width:2px;
    classDef docs fill:#E0BBE4,stroke:#957DAD,stroke-width:2px;
```
