How pablosanchez87/web_scrapping_chatbot works
Project Type & Purpose
• The project is a full-stack application for a chatbot that enriches language model outputs by retrieving domain-specific information stored in a vector database.
• It combines a Streamlit-based interactive user interface, scripts for scraping data from web pages and PDFs, and multiple components (embedding generation, vector storage using ChromaDB, and a language generation pipeline) to deliver context-relevant responses.
Main Components
a. User Interface:
– Streamlit App (app.py): Provides the chat interface and interaction logic.
b. Data Retrieval & Generation:
– Chat Logic & Response Generation: The component that processes user input, loads the embeddings from ChromaDB, and uses the ChatOpenAI (e.g., GPT-4) model to generate responses.
– Prompt Template: A dedicated part in app.py defining how context and user queries are formatted.
c. Embedding & Knowledge Base Management:
– Embedding Generation: update_embeddings.py (and similar scripts) is responsible for processing scraped texts, generating embeddings via the OpenAI 'text-embedding-ada-002' model, and storing/updating them in ChromaDB.
– ChromaDB Storage: Represented by the directory “chroma_db” (which includes binary files and a SQLite database) where the vector data is saved.
d. Data Ingestion / Web Scraping:
– Scraping Scripts: web_scrapping_pdfs_reports.py, web_scrapping_pdfs_news-insights.py, and web_scrapping_urls.py for extracting updated textual content from different sources.
e. Configuration and Deployment:
– Environment & Dependency Management: Files such as requirements.txt and .env configuration (for API keys) indicating integration with external APIs and dependency management.
– Deployment Notes: The notes_deploy.md file provides details for deploying the Streamlit app using Github, ensuring that the service is available to end users.
Relationships & Data Flow
– The user interacts with the Streamlit UI. When a query is submitted:
• The UI (app.py) takes the query and passes it to the response generation logic.
• This logic then calls load_embeddings to retrieve relevant vectors/documents from the ChromaDB.
• The retrieved context, along with the query, is forwarded to the ChatOpenAI model (GPT-4) via a constructed prompt.
• The generated answer is returned and displayed on the UI.
– Separately, the embedding generation process (via update_embeddings.py) works by:
• Using scraping scripts to extract content.
• Generating embeddings using OpenAIEmbeddings.
• Storing these embeddings into the ChromaDB (vector database) that is later queried by the chat system.
• User → UI (Streamlit)
• UI ↔ Backend (embedding loading, API call to ChatOpenAI, retrieval chain)
• External Sources/Data → Web Scraping Scripts → update_embeddings.py → ChromaDB
Architectural Patterns & Technologies
– This design follows a modular, separation-of-concerns approach with clear division between presentation (Streamlit UI), business logic (chat and retrieval functions in app.py), and data processing/management (scraping and embedding update scripts).
– Key technologies include:
• Python as the main programming language.
• Streamlit for the UI.
• OpenAI’s GPT-4 and text-embedding-ada-002 for natural language and embeddings.
• ChromaDB for vector storage and similarity-based retrieval.
• LangChain for orchestrating the query-to-response chain.
– Emphasize the Retrieval-Augmented Generation (RAG) mechanism, which integrates the content retrieval step with generative AI to produce context-informed answers.
Final Advice
– Be detailed; include every component and sub-module to capture the entire pipeline from data ingestion to response generation.
– Avoid over-simplification—capturing all relevant modules (even if it means more components) will best reflect the architecture of this RAG-based chatbot.