How mohitlunia/sol_rag_app works
Project Type and Purpose
• This is a Python-based Retrieval-Augmented Generation (RAG) prototype that:
– Extracts text from a list of URLs (“extractor” module)
– Builds or queries a retriever/embedding index (“rag_app/retriever.py”)
– Feeds retrieved passages and a QA prompt to an LLM (“rag_app/rag_app.py”)
• The top-level script (sol_rag_app.py) orchestrates the pipeline end-to-end.
File Structure
• extractor/
– extract.py: reads urls.txt, scrapes pages, writes pages_text.txt + failed_urls.txt
– urls.txt: seed URLs to process
• rag_app/
– retriever.py: loads pages_text.txt, builds or queries vector/index store
– prompts/qa_prompt.txt: prompt template for LLM queries
– rag_app.py: accepts user question, calls retriever, assembles prompt, sends to LLM API
• sol_rag_app.py: high-level entry point that ties extraction → retrieval → generation
• Data files (pages_text.txt, failed_urls.txt) act as simple local “data store”
• requirements.txt: lists dependencies (e.g., requests, BeautifulSoup, OpenAI SDK, FAISS or similar)
Key Components
a. Extractor Service
– Input: urls.txt
– Processing: extract.py (HTTP fetch → parse → text extraction)
– Outputs: pages_text.txt (raw documents), failed_urls.txt
b. Data Store
– Simple file storage for pages_text.txt
– Optionally a vector database (if retriever uses FAISS, Pinecone, etc.)
c. Retriever
– Input: pages_text.txt (or vector db) + user query
– Processing: embedding generation, similarity search
– Output: top-k relevant passages
d. RAG Application
– Inputs: retrieved passages + qa_prompt.txt + user question
– Processing: build final prompt, call LLM API (e.g. OpenAI GPT)
– Output: generated answer
e. Orchestration Script
– sol_rag_app.py: coordinates extraction, indexing, interactive QA loop
Relationships & Data Flows
1. urls.txt → Extractor → pages_text.txt + failed_urls.txt
2. pages_text.txt → Retriever → retrieved passages
3. retrieved passages + qa_prompt.txt + user question → RAG App → LLM API → answer
4. sol_rag_app.py ties steps 1–3 together in sequence or via CLI arguments
Architectural Patterns & Design Principles
• Separation of Concerns: Extraction, Retrieval, and Generation each in its own module
• Pipeline/Workflow Pattern: linear flow of data through discrete stages
• Extensibility: easy to swap out the embedding store (file vs. FAISS vs. external vector DB) or the LLM provider
• Configurable Prompts: kept in text file (qa_prompt.txt) for easy modification
Detailed Capture of Essential Architecture
• If needed, show internal calls (e.g., retriever.py → embed(text) → vector_index.search).
• Retry/failure paths (e.g., failed_urls.txt written if extract fails).