How heshi99/pdfreader-rag works
Project type and purpose
• This is a full-stack, document-centric Q&A system using a Retrieval-Augmented Generation (RAG) approach.
• Users upload PDF files via a web UI (Streamlit), the system extracts and indexes text, then answers queries by retrieving relevant chunks and running them through a Transformer QA model.
File structure
• Top-level files:
• backend.py – Flask app exposing REST endpoints
• frontend.py – Streamlit UI
• README.md, LICENSE
• Directories:
• uploads/ – raw PDF storage
• vector_db/ – FAISS index files and pickled embeddings
• faiss_index/ – (redundant/alternate) prebuilt FAISS indexes
• Artifacts: vectorstore.pkl, index.faiss, index.pkl
Key insights from the README
• Tech stack:
• Frontend: Streamlit
• Backend: Flask
• PDF handling: PyPDF2
• Embeddings: HuggingFaceEmbeddings via Langchain
• Similarity search: FAISS
• QA inference: Hugging Face Transformers
• Workflow stages spelled out: upload → extract & chunk → embed & index → retrieve & answer.
Detail level
• Drill down each layer: e.g., show that “Chunker & Embedder” uses Langchain to split text into 1,000-token windows, then calls HuggingFaceEmbeddings.
• Both batch paths (upload/index) and real-time paths (ask/retrieve/answer).