# rohitsharma0719/web-content-q-a-tool architecture

> Architecture diagram of the GitHub repository rohitsharma0719/web-content-q-a-tool, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rohitsharma0719/web-content-q-a-tool
- Repository: https://github.com/rohitsharma0719/web-content-q-a-tool

## Overview

##### Key components
- User Interface: A Streamlit-based front-end where users submit URLs and questions.
- Web Scraper & Content Extractor: Uses HTTP requests (via requests) and HTML parsing (via BeautifulSoup) to pull only the meaningful
elements from the target webpages.
- Text Pre-Processor: Splits the retrieved webpage content into manageable chunks.
- Embedding Generator: Converts text chunks into vector representations using a pre-trained Hugging Face sentence‐transformer model.
- Vector Store: FAISS database that stores the vectors for fast similarity look-up.
- Query Processor & Retriever: Compares the vector for the user’s input (question) to the stored vectors, retrieving the most relevant content segments.
- Answer Generator: Feeds the retrieved context into a lightweight LLaMA 2 GGUF model to generate the answer.

tags” using BeautifulSoup.
- Next, indicate that the extracted content goes to a Text Pre-Processor which splits content into chunks.
- Link the generated vectors to the FAISS Vector Store for efficient storage and later retrieval.
- On the query path, show the user’s question flowing from the UI into:
a. A vector transformation for the query,
b. Then to the FAISS Vector Store where similarity search retrieves the most relevant text segments.
- These retrieved segments are then provided to the LLaMA 2 GGUF-based Answer Generator.
- Finally, the answer is passed back to the Streamlit UI and displayed to the User.

##### Architectural Patterns and Technologies
- The project follows a modular pipeline architecture where each stage (scraping, chunking, embedding, retrieval, and answer generation) is clearly delineated, even if they are all integrated into one file.
- Separation of concerns:
– Data ingestion (web scraping)
– Data processing (chunking and embedding)
– Data retrieval (FAISS)
– Business logic (answer generation with LLaMA 2)
– Presentation layer (Streamlit UI)
- External libraries and technologies used:
– Streamlit for UI,
– requests for HTTP calls,
– BeautifulSoup for parsing HTML,
– Hugging Face transformers for embedding generation,
– FAISS for vector search,
– LLaMA 2 GGUF for answering queries.

##### Final Recommendations for the Principal Software Engineer
- That all components are distinctly labeled and represented so that even if they are implemented together (or are lightweight), their roles are clear in the overall architecture.
- Do not hesitate to split the architecture into “Frontend”, “Processing Pipeline”, and “Inference & Retrieval” layers for clarity, even if that adds components for conceptual purposes.

- Client (Browser/Streamlit UI)
- Web Scraping & HTML Content Extraction (using Requests and BeautifulSoup)
- Text Chunking and Preprocessing
- Embedding Generation (with Hugging Face's sentence-transformers)
- In-memory Vector Store (FAISS)
- Query Processing and Similarity Search
- LLaMA 2 GGUF Answer Generation
- Data flow back to the UI for user response

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        User["User"]
        UI["Streamlit UI"]:::ui
    end

    %% Processing Pipeline
    subgraph "Processing Pipeline"
        Scraper["Web Scraper & Content Extractor"]:::processing
        Chunker["Text Pre-Processor"]:::processing
        Embed["Embedding Generator"]:::processing
        Store["Vector Store (FAISS)"]:::storage
    end

    %% Inference & Retrieval
    subgraph "Inference & Retrieval"
        Retriever["Query Processor & Retriever"]:::processing
        AnswerGen["LLaMA 2 GGUF Answer Generator"]:::ai
    end

    %% Data Flow Connections
    User -->|"InitiateQ&A"| UI
    UI -->|"SubmitURL"| Scraper
    Scraper -->|"Extract<p>tags"| Chunker
    Chunker -->|"ChunkedText"| Embed
    Embed -->|"StoreVectors"| Store

    UI -->|"SubmitQuestion"| Retriever
    Retriever -->|"SimilaritySearch"| Store
    Store -->|"RelevantSegments"| Retriever
    Retriever -->|"ProvideContext"| AnswerGen
    AnswerGen -->|"DisplayAnswer"| UI

    %% Click Events for Components
    click UI "https://github.com/rohitsharma0719/web-content-q-a-tool/blob/main/app.py"
    click Scraper "https://github.com/rohitsharma0719/web-content-q-a-tool/blob/main/app.py"
    click Chunker "https://github.com/rohitsharma0719/web-content-q-a-tool/blob/main/app.py"
    click Embed "https://github.com/rohitsharma0719/web-content-q-a-tool/blob/main/app.py"
    click Store "https://github.com/rohitsharma0719/web-content-q-a-tool/blob/main/app.py"
    click Retriever "https://github.com/rohitsharma0719/web-content-q-a-tool/blob/main/app.py"
    click AnswerGen "https://github.com/rohitsharma0719/web-content-q-a-tool/blob/main/app.py"

    %% Styles
    classDef ui fill:#FFA500,stroke:#000,stroke-width:2px;
    classDef processing fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef storage fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef ai fill:#D8BFD8,stroke:#000,stroke-width:2px;
```
