# mohitlunia/sol_rag_app architecture

> Architecture diagram of the GitHub repository mohitlunia/sol_rag_app, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mohitlunia/sol_rag_app
- Repository: https://github.com/mohitlunia/sol_rag_app

## Overview

##### Project Type and Purpose
- This is a Python-based Retrieval-Augmented Generation (RAG) prototype that:
– Extracts text from a list of URLs (“extractor” module)
– Builds or queries a retriever/embedding index (“rag_app/retriever.py”)
– Feeds retrieved passages and a QA prompt to an LLM (“rag_app/rag_app.py”)
- The top-level script (sol_rag_app.py) orchestrates the pipeline end-to-end.

##### File Structure
- extractor/
– extract.py: reads urls.txt, scrapes pages, writes pages_text.txt + failed_urls.txt
– urls.txt: seed URLs to process
- rag_app/
– retriever.py: loads pages_text.txt, builds or queries vector/index store
– prompts/qa_prompt.txt: prompt template for LLM queries
– rag_app.py: accepts user question, calls retriever, assembles prompt, sends to LLM API
- sol_rag_app.py: high-level entry point that ties extraction → retrieval → generation
- Data files (pages_text.txt, failed_urls.txt) act as simple local “data store”
- requirements.txt: lists dependencies (e.g., requests, BeautifulSoup, OpenAI SDK, FAISS or similar)

##### Key Components
a. Extractor Service
– Input: urls.txt
– Processing: extract.py (HTTP fetch → parse → text extraction)
– Outputs: pages_text.txt (raw documents), failed_urls.txt
b. Data Store
– Simple file storage for pages_text.txt
– Optionally a vector database (if retriever uses FAISS, Pinecone, etc.)
c. Retriever
– Input: pages_text.txt (or vector db) + user query
– Processing: embedding generation, similarity search
– Output: top-k relevant passages
d. RAG Application
– Inputs: retrieved passages + qa_prompt.txt + user question
– Processing: build final prompt, call LLM API (e.g. OpenAI GPT)
– Output: generated answer
e. Orchestration Script
– sol_rag_app.py: coordinates extraction, indexing, interactive QA loop

##### Relationships & Data Flows
1. urls.txt → Extractor → pages_text.txt + failed_urls.txt
2. pages_text.txt → Retriever → retrieved passages
3. retrieved passages + qa_prompt.txt + user question → RAG App → LLM API → answer
4. sol_rag_app.py ties steps 1–3 together in sequence or via CLI arguments

##### Architectural Patterns & Design Principles
- Separation of Concerns: Extraction, Retrieval, and Generation each in its own module
- Pipeline/Workflow Pattern: linear flow of data through discrete stages
- Extensibility: easy to swap out the embedding store (file vs. FAISS vs. external vector DB) or the LLM provider
- Configurable Prompts: kept in text file (qa_prompt.txt) for easy modification

##### Detailed Capture of Essential Architecture
- If needed, show internal calls (e.g., retriever.py → embed(text) → vector_index.search).
- Retry/failure paths (e.g., failed_urls.txt written if extract fails).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration
    Orchestrator["sol_rag_app.py\nController/Orchestrator"]:::orchestrator

    %% Extraction Stage
    URLs["urls.txt"]:::file
    Extractor["Extractor Service\n(extract.py)"]:::processor
    PagesText[(pages_text.txt)]:::datastore
    FailedURLs[(failed_urls.txt)]:::datastore

    %% Retrieval Stage
    Retriever["Retriever\n(retriever.py)"]:::processor
    VectorStore[(Vector Store\nFAISS/Pinecone)]:::datastore

    %% Generation Stage
    PromptTemplate["Prompt Template\n(qa_prompt.txt)"]:::file
    RAGApp["RAG App\n(rag_app.py)"]:::processor
    ExternalLLM["LLM API\n(OpenAI GPT)"]:::external

    %% Workflow
    Orchestrator --> Extractor
    Extractor --> PagesText
    Extractor --> FailedURLs
    URLs --> Extractor

    PagesText --> Retriever
    Retriever --> VectorStore
    VectorStore --> Retriever

    Retriever --> RAGApp
    PromptTemplate --> RAGApp

    RAGApp --> ExternalLLM

    %% Click Events
    click Extractor "https://github.com/mohitlunia/sol_rag_app/blob/main/old_sol_rag_with_separate/extractor/extract.py"
    click URLs "https://github.com/mohitlunia/sol_rag_app/blob/main/old_sol_rag_with_separate/extractor/urls.txt"
    click PagesText "https://github.com/mohitlunia/sol_rag_app/blob/main/old_sol_rag_with_separate/extractor/pages_text.txt"
    click FailedURLs "https://github.com/mohitlunia/sol_rag_app/blob/main/old_sol_rag_with_separate/extractor/failed_urls.txt"
    click Retriever "https://github.com/mohitlunia/sol_rag_app/blob/main/old_sol_rag_with_separate/rag_app/retriever.py"
    click PromptTemplate "https://github.com/mohitlunia/sol_rag_app/blob/main/old_sol_rag_with_separate/rag_app/prompts/qa_prompt.txt"
    click RAGApp "https://github.com/mohitlunia/sol_rag_app/blob/main/old_sol_rag_with_separate/rag_app/rag_app.py"
    click Orchestrator "https://github.com/mohitlunia/sol_rag_app/blob/main/old_sol_rag_with_separate/sol_rag_app.py"
    click ExternalLLM callback
    click VectorStore callback

    %% Styles
    classDef processor fill:#bbdefb,stroke:#1e88e5,stroke-width:2px
    classDef file fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
    classDef datastore fill:#dcedc8,stroke:#558b2f,stroke-width:2px
    classDef external fill:#ffe0b2,stroke:#ef6c00,stroke-width:2px
    classDef orchestrator fill:#ffffff,stroke:#9e9e9e,stroke-dasharray: 5 5,stroke-width:2px
    class PagesText,FailedURLs,VectorStore datastore
    class Orchestrator orchestrator
```
