How gauravkhapekar/rag-pipeline-pdf-system works
Project Type and Purpose
• This is a Python‐based Retrieval‐Augmented Generation (RAG) pipeline specialized for ingesting PDF documents, chunking them, indexing embeddings in vector stores (Chroma and FAISS), retrieving relevant chunks, reranking them, invoking an LLM to generate responses, and producing final reports or a live Streamlit demo.
File Structure Analysis
• Top‐level scripts and configs:
• enhanced_rag_demo_app.py (Streamlit UI)
• run_app.sh
• config.yaml, .env.template
• requirements.txt
• Data directories:
• data/pdfs (raw PDFs)
• data/processed (intermediate JSON)
• data/embeddings/chroma_db & faiss_index (on‐disk vector stores)
• src/ folder modules by responsibility:
• data_processing/pdf_processor.py
• chunking/semantic_chunker.py
• indexing/ (vector store wrappers)
• retrieval/, reranking/, llm/ (integration with LLM API)
• reporting/docx_generator.py
• evaluation/benchmark_runner.py
README Insights
• The “Quick Start” shows a Streamlit app front end and plain‐Vanilla Python back end.
Main Components
a. Data Ingestion & Preprocessing
• PDF Processor (src/data_processing/pdf_processor.py)
• Semantic Chunker (src/chunking/semantic_chunker.py)
b. Embedding & Indexing Layer
• Embedding Generator (inside chunker or llm_integration)
• Chroma Vector Store (data/embeddings/chroma_db, managed by src/vector_stores/db_manager.py)
• FAISS Index (data/embeddings/faiss_index)
c. Retrieval & Reranking
• Retriever (src/retrieval) queries vector stores
• Reranker (src/reranking/reranker.py) boosts relevance
d. LLM Invocation
• LLM Integration (src/llm/llm_integration.py) calls external LLM API
e. Presentation & Reporting
• Streamlit Demo App (enhanced_rag_demo_app.py)
• DOCX Report Generator (src/reporting/docx_generator.py)
f. Evaluation Module
• Benchmark Runner (src/evaluation/benchmark_runner.py)
Component Interactions and Data Flow
1) Raw PDFs → PDF Processor → Chunks
2) Chunks → Embedding Generator → index stores (Chroma/FAISS)
3) User Query (UI) → Retriever → Candidate Chunks → Reranker
4) Top Chunks → LLM API → Generated Response
5) Response → UI display and/or DOCX Generator
• Configuration (.env, config.yaml) feeds into all modules for API keys, paths, and thresholds.
• Evaluation module taps into retrieval/reranking/LMM steps to measure performance.
Architectural Patterns & Principles
• Pipeline Pattern: clearly distinct stages (ingest → preprocess → store → retrieve → generate → report).
• Modular Design: each src/ subfolder is a plug‐and‐play component.
• Separation of Concerns: vector store management separated from chunking, retrieval, and LLM logic.
• External Service Integration: LLM API as a bounded context.
Technologies & Frameworks
• Python 3.x
• Streamlit (UI)
• Chroma DB & FAISS (vector storage)
• OpenAI (or other LLM provider) via llm_integration.py
• python‐docx or similar for report generation