# pablosanchez87/web_scrapping_chatbot architecture

> Architecture diagram of the GitHub repository pablosanchez87/web_scrapping_chatbot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pablosanchez87/web_scrapping_chatbot
- Repository: https://github.com/pablosanchez87/web_scrapping_chatbot

## Overview

##### Project Type & Purpose
- The project is a full-stack application for a chatbot that enriches language model outputs by retrieving domain-specific information stored in a vector database.
- It combines a Streamlit-based interactive user interface, scripts for scraping data from web pages and PDFs, and multiple components (embedding generation, vector storage using ChromaDB, and a language generation pipeline) to deliver context-relevant responses.

##### Main Components
a. User Interface:
– Streamlit App (app.py): Provides the chat interface and interaction logic.
b. Data Retrieval & Generation:
– Chat Logic & Response Generation: The component that processes user input, loads the embeddings from ChromaDB, and uses the ChatOpenAI (e.g., GPT-4) model to generate responses.
– Prompt Template: A dedicated part in app.py defining how context and user queries are formatted.
c. Embedding & Knowledge Base Management:
– Embedding Generation: update_embeddings.py (and similar scripts) is responsible for processing scraped texts, generating embeddings via the OpenAI 'text-embedding-ada-002' model, and storing/updating them in ChromaDB.
– ChromaDB Storage: Represented by the directory “chroma_db” (which includes binary files and a SQLite database) where the vector data is saved.
d. Data Ingestion / Web Scraping:
– Scraping Scripts: web_scrapping_pdfs_reports.py, web_scrapping_pdfs_news-insights.py, and web_scrapping_urls.py for extracting updated textual content from different sources.
e. Configuration and Deployment:
– Environment & Dependency Management: Files such as requirements.txt and .env configuration (for API keys) indicating integration with external APIs and dependency management.
– Deployment Notes: The notes_deploy.md file provides details for deploying the Streamlit app using Github, ensuring that the service is available to end users.

##### Relationships & Data Flow
– The user interacts with the Streamlit UI. When a query is submitted:
- The UI (app.py) takes the query and passes it to the response generation logic.
- This logic then calls load_embeddings to retrieve relevant vectors/documents from the ChromaDB.
- The retrieved context, along with the query, is forwarded to the ChatOpenAI model (GPT-4) via a constructed prompt.
- The generated answer is returned and displayed on the UI.
– Separately, the embedding generation process (via update_embeddings.py) works by:
- Using scraping scripts to extract content.
- Generating embeddings using OpenAIEmbeddings.
- Storing these embeddings into the ChromaDB (vector database) that is later queried by the chat system.
- User → UI (Streamlit)
- UI ↔ Backend (embedding loading, API call to ChatOpenAI, retrieval chain)
- External Sources/Data → Web Scraping Scripts → update_embeddings.py → ChromaDB

##### Architectural Patterns & Technologies
– This design follows a modular, separation-of-concerns approach with clear division between presentation (Streamlit UI), business logic (chat and retrieval functions in app.py), and data processing/management (scraping and embedding update scripts).
– Key technologies include:
- Python as the main programming language.
- Streamlit for the UI.
- OpenAI’s GPT-4 and text-embedding-ada-002 for natural language and embeddings.
- ChromaDB for vector storage and similarity-based retrieval.
- LangChain for orchestrating the query-to-response chain.
– Emphasize the Retrieval-Augmented Generation (RAG) mechanism, which integrates the content retrieval step with generative AI to produce context-informed answers.

##### Final Advice
– Be detailed; include every component and sub-module to capture the entire pipeline from data ingestion to response generation.
– Avoid over-simplification—capturing all relevant modules (even if it means more components) will best reflect the architecture of this RAG-based chatbot.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction Layer
    subgraph "User Interaction Layer"
        User["User"]:::external
        UI["User Interface / Streamlit App"]:::ui
    end

    %% Chat/Response Generation Pipeline
    subgraph "Chat/Response Generation Pipeline"
        Chat["Chat/Response Generation Module"]:::backend
        Prompt["Prompt Template"]:::backend
        API["External APIs (OpenAI)"]:::external
    end

    %% Data Ingestion & Embedding Pipeline
    subgraph "Data Ingestion & Embedding Pipeline"
        subgraph "Web Scraping Modules"
            ScrapeNews["web_scrapping_pdfs_news-insights.py"]:::data
            ScrapeReports["web_scrapping_pdfs_reports.py"]:::data
            ScrapeURLs["web_scrapping_urls.py"]:::data
        end
        Embed["Embedding Generation Module"]:::data
    end

    %% Vector Storage Layer
    subgraph "Vector Storage Layer"
        DB["ChromaDB (Vector Database)"]:::database
        SQL["SQLite (chroma_db/chroma.sqlite3)"]:::database
    end

    %% Configuration & Deployment Components
    subgraph "Configuration & Deployment"
        ConfigReq["Dependency Management (requirements.txt)"]:::config
        DeployNotes["Deployment Notes (notes_deploy.md)"]:::config
    end

    %% Data Flow Connections
    User -->|"uses"| UI
    UI -->|"sends_query"| Chat
    Chat -->|"formats_with"| Prompt
    Chat -->|"retrieves_context"| DB
    Chat -->|"calls_API"| API
    ScrapeNews -->|"provides_data"| Embed
    ScrapeReports -->|"provides_data"| Embed
    ScrapeURLs -->|"provides_data"| Embed
    Embed -->|"updates"| DB
    ConfigReq -.-> UI
    ConfigReq -.-> Chat
    ConfigReq -.-> Embed
    DeployNotes -.-> UI
    DeployNotes -.-> Chat
    DeployNotes -.-> Embed

    %% Click Events
    click UI "https://github.com/pablosanchez87/web_scrapping_chatbot/blob/main/app.py"
    click Chat "https://github.com/pablosanchez87/web_scrapping_chatbot/blob/main/app.py"
    click Embed "https://github.com/pablosanchez87/web_scrapping_chatbot/blob/main/update_embeddings.py"
    click DB "https://github.com/pablosanchez87/web_scrapping_chatbot/tree/main/chroma_db"
    click ScrapeNews "https://github.com/pablosanchez87/web_scrapping_chatbot/blob/main/web_scrapping_pdfs_news-insights.py"
    click ScrapeReports "https://github.com/pablosanchez87/web_scrapping_chatbot/blob/main/web_scrapping_pdfs_reports.py"
    click ScrapeURLs "https://github.com/pablosanchez87/web_scrapping_chatbot/blob/main/web_scrapping_urls.py"
    click ConfigReq "https://github.com/pablosanchez87/web_scrapping_chatbot/blob/main/requirements.txt"
    click DeployNotes "https://github.com/pablosanchez87/web_scrapping_chatbot/blob/main/notes_deploy.md"

    %% Styles
    classDef ui fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef backend fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef data fill:#D5F5E3,stroke:#27AE60,stroke-width:2px;
    classDef database fill:#F5CBA7,stroke:#E67E22,stroke-width:2px;
    classDef external fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px;
    classDef config fill:#FADBD8,stroke:#C0392B,stroke-width:2px;
```
