How tahanadeemkhan/web_integration-main works
Project type and purpose
• This is a Streamlit‑based web application that serves as an interactive chatbot for exploring and querying the content of any website. It loads web page text, chunks it, stores embeddings in a vector database (Chroma), and uses LangChain plus Google’s Gemini model to retrieve and generate answers.
File structure
• Top‑level files:
• .devcontainer/devcontainer.json – development container configuration
• README.md – project overview and setup instructions
• app.py – the entire Streamlit application code (UI + backend logic)
• requirements.txt – Python dependencies list
• No separate frontend or backend directories; everything resides in a single monolithic script.
Key insights from README
• Core features: web content loading, text splitting, embedding storage with Chroma, chat interface, contextual response filtering.
• Tech stack: Python 3.8+, Streamlit, LangChain, HuggingFace transformers, Chroma vector DB, Google Generative AI (Gemini).
Main components
a. Streamlit UI layer
• Renders input field for website URL
• Displays chat interface for user questions and answers
b. Content Ingestion Pipeline
• Web crawler / scraper module (requests + BeautifulSoup or similar)
• Text splitter (e.g., LangChain’s RecursiveCharacterTextSplitter)
c. Vector Storage & Retrieval
• Chroma vector database for storing document embeddings
• Retriever component from LangChain to query similar chunks
d. LLM / Generation Service
• Google Gemini model accessed via Google Generative AI API
• LangChain’s LLM wrapper orchestrates prompt template and API calls
e. Environment & Config
• .env file for GOOGLE_API_KEY
• Devcontainer for consistent development environment
Relationships and interactions
• User submits a website URL → Streamlit triggers content ingestion
• Ingestion service fetches HTML → text splitter breaks into chunks → embeddings are computed via LangChain + embedding model → Chroma stores embeddings
• User asks a question in chat → Retriever queries Chroma for relevant chunks → retrieved context + user query passed to LangChain prompt → Streamlit displays Gemini’s response
• All data flows are synchronous within the single app process
Architectural patterns / design principles
• Retrieval‑Augmented Generation (RAG) pattern
• Modular pipeline: ingestion, embedding, storage, retrieval, generation
• Single‑process, event‑driven flow managed by Streamlit callbacks
Technology call‑outs
• Streamlit: UI and app server
• LangChain: orchestration of text splitting, embeddings, retrieval, LLM calls
• Chroma: lightweight vector store
• Google Generative AI API: hosting Gemini model
• Python environment (.devcontainer, .env)
Elements
• Environment/config sidecar (API keys, devcontainer)