How arjunsah21/inotebook works
Project type and purpose
• This is a Retrieval-Augmented Generation (RAG) system implemented in Python.
• It provides a Streamlit-based UI for ingesting PDF documents, indexing their contents in Milvus, and running OpenAI GPT-4 over retrieved chunks to answer user questions.
File structure
• Top-level scripts:
• app.py – Streamlit application (frontend + orchestrator)
• document_processor.py – PDF/text/table/image extraction (uses PyPDF2 & Tesseract)
• vector_store.py – Milvus client (pymilvus) for indexing & retrieval
• rag_chain.py – LangChain RAG orchestration + conversation memory
• docker-compose.yml – Spins up Milvus standalone container
• test_milvus.py – Connectivity/indexing tests for the vector store
• Config files: .env.example, requirements.txt
README details
• PDF processing, Milvus vector database, OpenAI GPT-4, Streamlit UI
• Setup: pip install, .env for API keys, Docker-Compose for Milvus, Tesseract OCR
Main components & interactions
a. Components
• Streamlit UI (app.py)
• PDF Processor (document_processor.py)
• Vector Store (vector_store.py → Milvus container)
• RAG Chain (rag_chain.py)
• External Services: OpenAI GPT-4 API, Tesseract OCR engine
• Configuration (.env)
b. Relationships & Data Flow
1. User uploads PDF in Streamlit UI → app.py
2. app.py calls document_processor.py → extracts text/tables/images → chunks content
3. document_processor.py calls vector_store.py → generates embeddings → indexes in Milvus
4. On user query, app.py calls rag_chain.py → vector_store.py retrieves top-k chunks → rag_chain.py calls OpenAI API (with conversation memory) → returns answer to app.py → displayed in UI
c. Architectural patterns
• Pipeline pattern: Ingestion → Indexing → Retrieval → Generation
• Separation of concerns: each Python module has a single responsibility
• External microservice: Milvus runs separately (Docker Compose)
d. Key technologies
• Streamlit (UI)
• PyPDF2 & Tesseract OCR (PDF extraction)
• pymilvus (Milvus client)
• LangChain (RAG orchestration + memory)
• OpenAI Python SDK (GPT-4 calls)
• Docker Compose (Milvus deployment)
Detailed component callouts
• Test_milvus.py off to the side as a verification utility feeding into vector_store.py