# gauravkhapekar/rag-pipeline-pdf-system architecture

> Architecture diagram of the GitHub repository gauravkhapekar/rag-pipeline-pdf-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gauravkhapekar/rag-pipeline-pdf-system
- Repository: https://github.com/gauravkhapekar/rag-pipeline-pdf-system

## Overview

##### Project Type and Purpose
- This is a Python‐based Retrieval‐Augmented Generation (RAG) pipeline specialized for ingesting PDF documents, chunking them, indexing embeddings in vector stores (Chroma and FAISS), retrieving relevant chunks, reranking them, invoking an LLM to generate responses, and producing final reports or a live Streamlit demo.

##### File Structure Analysis
- Top‐level scripts and configs:
- enhanced_rag_demo_app.py (Streamlit UI)
- run_app.sh
- config.yaml, .env.template
- requirements.txt
- Data directories:
- data/pdfs (raw PDFs)
- data/processed (intermediate JSON)
- data/embeddings/chroma_db & faiss_index (on‐disk vector stores)
- src/ folder modules by responsibility:
- data_processing/pdf_processor.py
- chunking/semantic_chunker.py
- indexing/ (vector store wrappers)
- retrieval/, reranking/, llm/ (integration with LLM API)
- reporting/docx_generator.py
- evaluation/benchmark_runner.py

##### README Insights
- The “Quick Start” shows a Streamlit app front end and plain‐Vanilla Python back end.

##### Main Components
a. Data Ingestion & Preprocessing
- PDF Processor (src/data_processing/pdf_processor.py)
- Semantic Chunker (src/chunking/semantic_chunker.py)
b. Embedding & Indexing Layer
- Embedding Generator (inside chunker or llm_integration)
- Chroma Vector Store (data/embeddings/chroma_db, managed by src/vector_stores/db_manager.py)
- FAISS Index (data/embeddings/faiss_index)
c. Retrieval & Reranking
- Retriever (src/retrieval) queries vector stores
- Reranker (src/reranking/reranker.py) boosts relevance
d. LLM Invocation
- LLM Integration (src/llm/llm_integration.py) calls external LLM API
e. Presentation & Reporting
- Streamlit Demo App (enhanced_rag_demo_app.py)
- DOCX Report Generator (src/reporting/docx_generator.py)
f. Evaluation Module
- Benchmark Runner (src/evaluation/benchmark_runner.py)

##### Component Interactions and Data Flow
1) Raw PDFs → PDF Processor → Chunks
2) Chunks → Embedding Generator → index stores (Chroma/FAISS)
3) User Query (UI) → Retriever → Candidate Chunks → Reranker
4) Top Chunks → LLM API → Generated Response
5) Response → UI display and/or DOCX Generator
- Configuration (.env, config.yaml) feeds into all modules for API keys, paths, and thresholds.
- Evaluation module taps into retrieval/reranking/LMM steps to measure performance.

##### Architectural Patterns & Principles
- Pipeline Pattern: clearly distinct stages (ingest → preprocess → store → retrieve → generate → report).
- Modular Design: each src/ subfolder is a plug‐and‐play component.
- Separation of Concerns: vector store management separated from chunking, retrieval, and LLM logic.
- External Service Integration: LLM API as a bounded context.

##### Technologies & Frameworks
- Python 3.x
- Streamlit (UI)
- Chroma DB & FAISS (vector storage)
- OpenAI (or other LLM provider) via llm_integration.py
- python‐docx or similar for report generation

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        PDF["PDF Processor"]:::processing
        Chunker["Semantic Chunker"]:::processing
    end

    %% Embedding & Indexing Layer
    subgraph "Embedding & Indexing Layer"
        EmbGen["Embedding Generator"]:::processing
        DBMgr["Vector Store Manager"]:::processing
        subgraph "Vector Stores"
            Chroma["Chroma Vector Store"]:::datastore
            FAISS["FAISS Index"]:::datastore
        end
    end

    %% Retrieval & Reranking
    subgraph "Retrieval & Reranking"
        Retriever["Retriever"]:::processing
        Reranker["Reranker"]:::processing
    end

    %% LLM Invocation
    subgraph "LLM Invocation"
        LLM["LLM Integration"]:::external
    end

    %% Presentation & Reporting
    subgraph "Presentation & Reporting"
        UI["Streamlit Demo App"]:::presentation
        DOCX["DOCX Report Generator"]:::presentation
    end

    %% Evaluation
    subgraph "Evaluation Module"
        Bench["Benchmark Runner"]:::processing
    end

    %% Configuration and Scripts
    Config["config.yaml"]:::config
    Env[".env.template"]:::config
    Run["run_app.sh"]:::config
    Req["requirements.txt"]:::config

    %% Data Flows
    PDF -->|"processed text"| Chunker
    Chunker -->|"text chunks"| EmbGen
    EmbGen --> DBMgr
    DBMgr -->|"embeddings"| Chroma
    DBMgr -->|"embeddings"| FAISS

    UI -->|"query"| Retriever
    Retriever -->|"candidate chunks"| Reranker
    Reranker -->|"top chunks"| LLM
    LLM -->|"response"| UI
    LLM -->|"response"| DOCX

    %% Evaluation taps
    Retriever --> Bench
    Reranker --> Bench
    LLM --> Bench

    %% Config and environment
    Config -.-> PDF
    Config -.-> Chunker
    Config -.-> DBMgr
    Config -.-> Retriever
    Config -.-> Reranker
    Config -.-> LLM
    Config -.-> UI
    Config -.-> DOCX
    Env -.-> PDF
    Env -.-> Chunker
    Env -.-> DBMgr
    Env -.-> Retriever
    Env -.-> Reranker
    Env -.-> LLM

    %% Startup and requirements
    Run --> UI
    Req -.-> PDF
    Req -.-> Chunker
    Req -.-> EmbGen
    Req -.-> DBMgr
    Req -.-> Retriever
    Req -.-> Reranker
    Req -.-> LLM
    Req -.-> UI
    Req -.-> DOCX

    %% Click Events
    click PDF "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/blob/main/src/data_processing/pdf_processor.py"
    click Chunker "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/blob/main/src/chunking/semantic_chunker.py"
    click Chroma "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/tree/main/data/embeddings/chroma_db/"
    click FAISS "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/tree/main/data/embeddings/faiss_index/"
    click DBMgr "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/blob/main/src/vector_stores/db_manager.py"
    click Retriever "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/blob/main/src/retrieval/__init__.py"
    click Reranker "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/blob/main/src/reranking/reranker.py"
    click LLM "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/blob/main/src/llm/llm_integration.py"
    click UI "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/blob/main/enhanced_rag_demo_app.py"
    click DOCX "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/blob/main/src/reporting/docx_generator.py"
    click Bench "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/blob/main/src/evaluation/benchmark_runner.py"
    click Config "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/blob/main/config.yaml"
    click Env "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/blob/main/.env.template"
    click Run "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/blob/main/run_app.sh"
    click Req "https://github.com/gauravkhapekar/rag-pipeline-pdf-system/blob/main/requirements.txt"

    %% Styles
    classDef processing fill:#e0f7e9,stroke:#2f8f4e,color:#2f8f4e
    classDef datastore fill:#e0f0ff,stroke:#1f5fbf,color:#1f5fbf
    classDef external fill:#fff4e0,stroke:#ff8f1f,color:#ff8f1f
    classDef presentation fill:#fffbe0,stroke:#cfae1f,color:#cfae1f
    classDef config fill:#f0f0f0,stroke:#888888,color:#555555
```
