# tahanadeemkhan/web_integration-main architecture

> Architecture diagram of the GitHub repository tahanadeemkhan/web_integration-main, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tahanadeemkhan/web_integration-main
- Repository: https://github.com/tahanadeemkhan/web_integration-main

## Overview

##### Project type and purpose
- This is a Streamlit‑based web application that serves as an interactive chatbot for exploring and querying the content of any website. It loads web page text, chunks it, stores embeddings in a vector database (Chroma), and uses LangChain plus Google’s Gemini model to retrieve and generate answers.

##### File structure
- Top‑level files:
- .devcontainer/devcontainer.json – development container configuration
- README.md – project overview and setup instructions
- app.py – the entire Streamlit application code (UI + backend logic)
- requirements.txt – Python dependencies list
- No separate frontend or backend directories; everything resides in a single monolithic script.

##### Key insights from README
- Core features: web content loading, text splitting, embedding storage with Chroma, chat interface, contextual response filtering.
- Tech stack: Python 3.8+, Streamlit, LangChain, HuggingFace transformers, Chroma vector DB, Google Generative AI (Gemini).

##### Main components
a. Streamlit UI layer
- Renders input field for website URL
- Displays chat interface for user questions and answers
b. Content Ingestion Pipeline
- Web crawler / scraper module (requests + BeautifulSoup or similar)
- Text splitter (e.g., LangChain’s RecursiveCharacterTextSplitter)
c. Vector Storage & Retrieval
- Chroma vector database for storing document embeddings
- Retriever component from LangChain to query similar chunks
d. LLM / Generation Service
- Google Gemini model accessed via Google Generative AI API
- LangChain’s LLM wrapper orchestrates prompt template and API calls
e. Environment & Config
- .env file for GOOGLE_API_KEY
- Devcontainer for consistent development environment

##### Relationships and interactions
- User submits a website URL → Streamlit triggers content ingestion
- Ingestion service fetches HTML → text splitter breaks into chunks → embeddings are computed via LangChain + embedding model → Chroma stores embeddings
- User asks a question in chat → Retriever queries Chroma for relevant chunks → retrieved context + user query passed to LangChain prompt → Streamlit displays Gemini’s response
- All data flows are synchronous within the single app process

##### Architectural patterns / design principles
- Retrieval‑Augmented Generation (RAG) pattern
- Modular pipeline: ingestion, embedding, storage, retrieval, generation
- Single‑process, event‑driven flow managed by Streamlit callbacks

##### Technology call‑outs
- Streamlit: UI and app server
- LangChain: orchestration of text splitting, embeddings, retrieval, LLM calls
- Chroma: lightweight vector store
- Google Generative AI API: hosting Gemini model
- Python environment (.devcontainer, .env)

##### Elements
- Environment/config sidecar (API keys, devcontainer)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Streamlit Layer
    subgraph "Streamlit Layer"
        UI["Streamlit UI (Input Field & Chat Interface)"]:::ui
        Server["Streamlit Server Logic"]:::ui
    end

    %% Content Ingestion Pipeline
    subgraph "Ingestion Pipeline"
        Ingestion["Web Scraper / HTML Loader"]:::pipeline
        Splitter["Text Splitter"]:::pipeline
        Embed["Embedding Model"]:::pipeline
    end

    %% Vector Storage & Retrieval
    subgraph "Vector Storage & Retrieval"
        DB["Chroma Vector DB"]:::db
        Retriever["LangChain Retriever"]:::service
    end

    %% LLM Service
    subgraph "LLM / Generation Service"
        Gemini["Google Gemini API"]:::external
    end

    %% Environment & Config
    subgraph "Environment & Configuration"
        DevCont[".devcontainer/devcontainer.json"]:::config
        Req["requirements.txt"]:::config
        Readme["README.md"]:::config
        GitIgnore[".gitignore"]:::config
    end

    %% Data Flows
    UI -->|"/submit URL"| Server
    Server -->|"raw HTML"| Ingestion
    Ingestion -->|"text"| Splitter
    Splitter -->|"text chunks"| Embed
    Embed -->|"embeddings"| DB
    UI -->|"/ask query"| Retriever
    Retriever -->|"relevant chunks & query"| Gemini
    Gemini -->|"generated answer"| UI
    DB -->|"vector search results"| Retriever

    %% Click Events
    click UI "https://github.com/tahanadeemkhan/web_integration-main/blob/main/app.py"
    click Server "https://github.com/tahanadeemkhan/web_integration-main/blob/main/app.py"
    click Ingestion "https://github.com/tahanadeemkhan/web_integration-main/blob/main/app.py"
    click Splitter "https://github.com/tahanadeemkhan/web_integration-main/blob/main/app.py"
    click Embed "https://github.com/tahanadeemkhan/web_integration-main/blob/main/app.py"
    click Retriever "https://github.com/tahanadeemkhan/web_integration-main/blob/main/app.py"
    click Gemini "https://github.com/tahanadeemkhan/web_integration-main/blob/main/app.py"
    click DevCont "https://github.com/tahanadeemkhan/web_integration-main/blob/main/.devcontainer/devcontainer.json"
    click Req "https://github.com/tahanadeemkhan/web_integration-main/blob/main/requirements.txt"
    click Readme "https://github.com/tahanadeemkhan/web_integration-main/blob/main/README.md"
    click GitIgnore "https://github.com/tahanadeemkhan/web_integration-main/blob/main/.gitignore"

    %% Styles
    classDef ui fill:#D0E8FF,stroke:#0288D1
    classDef pipeline fill:#FFF4D0,stroke:#FBC02D
    classDef db fill:#D0F8E0,stroke:#388E3C
    classDef service fill:#FFE0B2,stroke:#F57C00
    classDef external fill:#FCE4EC,stroke:#D81B60
    classDef config fill:#E8EAF6,stroke:#5C6BC0
```
