How srkchaithanya/chatpdf works
Project Type and Purpose
• This is a single-page, full-stack–style Streamlit application (“ChatPDF”) for interactive Q&A over PDF documents.
• It combines a lightweight web UI (Streamlit) with a server-side pipeline: PDF ingestion → text chunking → embedding → vector storage → retrieval + LLM (Google Gemini)–based QA.
File Structure
• Top-level files only:
– app.py: Entrypoint; holds UI code, orchestrates the pipeline.
– requirements.txt: Lists dependencies (Streamlit, PyPDF2, LangChain, FAISS, Google AI SDK).
– README.md: Explains overall architecture and usage.
• No nested folders—everything lives in app.py, but conceptually it splits into modules/pipeline stages.
README for Insights
• Confirms a 5-step pipeline: PDF Upload & Extraction → Text Chunking → Embedding → FAISS Storage → Question Answering.
• External dependencies: Google Generative AI (Gemini) for embeddings + LLM, FAISS local vector index.
• UI is a Streamlit sidebar + main chat area.
Main Components
a. Streamlit UI Layer
• File-uploader widget (PDF input)
b. PDF Processing Module
• PyPDF2 Text Extractor
• LangChain RecursiveCharacterTextSplitter
c. Embedding Service
• GoogleGenerativeAIEmbeddings (calls Google AI API)
d. Vector Store
• FAISS index (persisted on local disk under faiss_index/)
e. Question-Answering Chain
• LangChain load_qa_chain using gemini-2.0-flash LLM
• Prompt template enforcing context-only answers
f. Configuration Store
• .env / environment variables (GOOGLE_API_KEY)
Relationships and Data Flow
1) User uploads PDF(s) via Streamlit UI →
2) app.py invokes PDF Processor → raw text
3) Text Splitter → list of chunks
4) Embedding Service → embedding vectors
5) Store embeddings in FAISS (local FS)
6) On user question:
a) Query UI →
b) FAISS similarity search → top-K chunks →
c) Pass chunks + question → QA Chain (Google LLM) →
d) Return answer → UI
Also: Embedding Service and QA Chain each call out to Google AI REST endpoints.
Architectural Patterns & Design Principles
• Pipeline pattern: clear stages of data transformation.
• Separation of concerns: UI vs. PDF ingestion vs. vector store vs. LLM.
• Externalize secrets via .env.
• Local persistence of index for fast retrieval (caching).