# heshi99/pdfreader-rag architecture

> Architecture diagram of the GitHub repository heshi99/pdfreader-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/heshi99/pdfreader-rag
- Repository: https://github.com/heshi99/pdfreader-rag

## Overview

##### Project type and purpose
- This is a full-stack, document-centric Q&A system using a Retrieval-Augmented Generation (RAG) approach.
- Users upload PDF files via a web UI (Streamlit), the system extracts and indexes text, then answers queries by retrieving relevant chunks and running them through a Transformer QA model.

##### File structure
- Top-level files:
- backend.py – Flask app exposing REST endpoints
- frontend.py – Streamlit UI
- README.md, LICENSE
- Directories:
- uploads/ – raw PDF storage
- vector_db/ – FAISS index files and pickled embeddings
- faiss_index/ – (redundant/alternate) prebuilt FAISS indexes
- Artifacts: vectorstore.pkl, index.faiss, index.pkl

##### Key insights from the README
- Tech stack:
- Frontend: Streamlit
- Backend: Flask
- PDF handling: PyPDF2
- Embeddings: HuggingFaceEmbeddings via Langchain
- Similarity search: FAISS
- QA inference: Hugging Face Transformers
- Workflow stages spelled out: upload → extract & chunk → embed & index → retrieve & answer.

##### Detail level
- Drill down each layer: e.g., show that “Chunker & Embedder” uses Langchain to split text into 1,000-token windows, then calls HuggingFaceEmbeddings.
- Both batch paths (upload/index) and real-time paths (ask/retrieve/answer).

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Frontend Layer"
        UI["Streamlit UI"]:::frontend
    end

    subgraph "Backend Layer"
        API["Flask API"]:::service

        subgraph "Ingestion Pipeline"
            ETL["Text-Extraction Service\n(PyPDF2)"]:::service
            Chunk["Chunker & Embedding Service\n(Langchain + HFEmbeddings)"]:::ml
        end

        subgraph "Query Pipeline"
            Retrieve["FAISS Vector Index\n(vector_db/index.faiss\nvector_db/index.pkl)"]:::storage
            QA["QA Inference Engine\n(Transformers)"]:::ml
        end
    end

    FileStore["PDF File Store\nuploads/"]:::storage
    IndexFiles["Serialized Vector Store\nvectorstore.pkl"]:::storage
    AltIndex["Prebuilt FAISS Index\nfaiss_index/"]:::storage

    UI -->|"upload PDF (POST /upload)"| API
    API -->|"save PDF"| FileStore
    API -->|"extract text"| ETL
    ETL -->|"text chunks"| Chunk
    Chunk -->|"embed & update index"| Retrieve
    Chunk -->|"serialize embeddings"| IndexFiles

    UI -->|"ask question (POST /ask)"| API
    API -->|"retrieve vectors"| Retrieve
    Retrieve -->|"top N chunks"| QA
    QA -->|"answer"| API
    API -->|"return answer"| UI

    API -->|"load prebuilt index"| AltIndex

    click UI "https://github.com/heshi99/pdfreader-rag/blob/master/frontend.py"
    click API "https://github.com/heshi99/pdfreader-rag/blob/master/backend.py"
    click FileStore "https://github.com/heshi99/pdfreader-rag/tree/master/uploads/"
    click ETL "https://github.com/heshi99/pdfreader-rag/blob/master/backend.py"
    click Chunk "https://github.com/heshi99/pdfreader-rag/blob/master/backend.py"
    click Retrieve "https://github.com/heshi99/pdfreader-rag/blob/master/vector_db/index.faiss"
    click Retrieve "https://github.com/heshi99/pdfreader-rag/blob/master/vector_db/index.pkl"
    click IndexFiles "https://github.com/heshi99/pdfreader-rag/blob/master/vectorstore.pkl"
    click AltIndex "https://github.com/heshi99/pdfreader-rag/tree/master/faiss_index/"
    click QA "https://github.com/heshi99/pdfreader-rag/blob/master/backend.py"

    classDef frontend fill:#87CEFA,stroke:#000,stroke-width:1px;
    classDef service fill:#4CAF50,stroke:#000,stroke-width:1px;
    classDef storage fill:#A9A9A9,stroke:#000,stroke-width:1px;
    classDef ml fill:#FFA500,stroke:#000,stroke-width:1px;
```
