How raisatahasen/rag works
Project type and purpose
• This is a full-stack, bilingual Retrieval-Augmented Generation (RAG) pipeline.
• It ingests a Bangla PDF document, chunks and embeds its content, stores vectors in a vector database, retrieves relevant chunks on user queries (Bangla or English), and then generates context-aware answers via an LLM exposed over a simple web API with a chat UI.
File structure
Top level:
• .env, requirements.txt – environment variables and dependencies
• README.md, Setup Guide.pdf, RAG_Q_A.pdf – docs and assessment materials
• api.py, chat_ui.html, main.py – API and UI entry points
• preprocessor.py, embedder.py, vector_store.py, retriever.py, generator.py – core pipeline stages
• data/ – contains source PDF (HSC26-Bangla1st-Paper.pdf)
• RAG_Q_A.pdf, Screenshot_of_DemoChat – reference/test files
Additional insights from README
• Uses Flask + Flask-CORS for the web API.
• Data pipeline leverages PyMuPDF, LlamaIndex for chunking, SentenceTransformers for embeddings (paraphrase-multilingual-mpnet-base-v2), ChromaDB (via LangChain) for vector storage, and OpenRouter to call an LLM (Anthropic/Claude-3).
• Session-aware API: clients POST {query, session_id} to /chat; context stored in memory.
Detail
• Environment config (.env) feeding API & services with API keys.
• Optional annotation of background scripts (e.g., a one-time main.py or vector_store.py job to build the index).
• Interfaces (HTTP REST) and data formats (JSON for /chat).