How jatiink/rag-qa-generator works
Project Type and Purpose
• This is a single-page Streamlit web application (“full-stack” in the sense of UI + backend pipeline) whose purpose is to:
• Ingest a user-uploaded PDF (or TXT)
• Split and embed the text into a FAISS vector store
• Use RetrievalQA with an LLM (llama2) to generate MCQs, True/False, and one-word-answer questions
• Results in the browser and persist them in a TXT file
Main Components
a. Presentation Layer (Frontend)
– Streamlit Web UI (app.py)
– File uploader widget
– Question-type selector
– Display area for generated questions
b. Application/Processing Layer (Backend pipeline inside app.py)
1) Document Loader: PyPDFLoader or TXT loader
2) Text Splitter: RecursiveCharacterTextSplitter
3) Embeddings: HuggingFaceEmbeddings (local or cloud)
4) Vector Store: FAISS index (in-memory or on-disk)
5) RetrievalQA: orchestrates query → vector search → prompt building → LLM inference
6) LLM Inference: llama2 engine via Ollama runtime
c. Persistence
– Input storage: uploaded PDF/TXT (e.g., Big Mac Index.pdf)
– Output store: Generated_Questions.txt
d. External Dependencies
– Ollama daemon (serving llama2 model)
– Python libraries (requirements.txt)
Relationships & Data Flow
1) Browser → Streamlit App: user uploads a PDF/TXT and selects question type
2) Streamlit App → Document Loader: loads raw text
3) Document Loader → Text Splitter: breaks text into chunks
4) Text Splitter → Embedding Engine: transforms chunks into vectors
5) Embeddings → FAISS Vector Store: builds/searchable index
6) User Query (RetrievalQA) → FAISS: retrieves top-k relevant chunks
7) Retrieved Chunks → Prompt → llama2 via Ollama → generates questions
8) Streamlit App → Browser: displays questions
9) Streamlit App → File System: writes Generated_Questions.txt
Architectural Patterns & Design Principles
• Modular pipeline: clearly separated ETL (load, split, embed, store) and QA (retrieve, prompt, infer) phases
• Single-container monolith: all components run inside one Python process, but logically separated
• Reuse of open-source building blocks (LangChain patterns)
• Stateless UI with stateful storage (FAISS + TXT file)