# debanjan110d/ai-web-scraper architecture

> Architecture diagram of the GitHub repository debanjan110d/ai-web-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/debanjan110d/ai-web-scraper
- Repository: https://github.com/debanjan110d/ai-web-scraper

## Overview

Ai-Web-Scraper is a small Streamlit application for fetching a website, cleaning its page text, and preparing it for description-driven extraction. The implemented UI calls the scraping and content-processing functions, while parse.py independently defines Ollama-backed extraction over text chunks. The sampled main.py ends after chunking and does not call that parser or display parsed results, so that connection is not inferred. The README documents a results table and automatic Ollama use, but these are not supported by the sampled implementation.

Main entry points: [`main.py`](https://github.com/debanjan110d/ai-web-scraper/blob/main/main.py) (Scraper UI), [`scrape.py`](https://github.com/debanjan110d/ai-web-scraper/blob/main/scrape.py) (Website Scraper), [`parse.py`](https://github.com/debanjan110d/ai-web-scraper/blob/main/parse.py) (Ollama Parser)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User Interface"]
  node_streamlit["Scraper UI<br/>[main.py]"]
end

subgraph group_content["Page Processing"]
  node_scrape["Website Scraper<br/>[scrape.py]"]
  node_extract_body["Body Extraction<br/>[scrape.py]"]
  node_clean_body["Text Cleaning<br/>[scrape.py]"]
  node_split_dom["Text Chunking<br/>[scrape.py]"]
end

subgraph group_extraction["AI Extraction"]
  node_parser["Ollama Parser<br/>[parse.py]"]
end

node_user(("User"))
node_browser["Chrome Browser"]
node_website["Target Website"]
node_ollama["Ollama Model"]

node_user -->|"enters URL"| node_streamlit
node_user -->|"enters description"| node_streamlit
node_streamlit -->|"calls scrape"| node_scrape
node_scrape -->|"drives browser"| node_browser
node_browser -->|"loads page"| node_website
node_scrape -->|"returns HTML"| node_streamlit
node_streamlit -->|"calls extraction"| node_extract_body
node_streamlit -->|"calls cleaning"| node_clean_body
node_streamlit -->|"calls chunking"| node_split_dom
node_parser -->|"invokes model"| node_ollama

click node_streamlit "https://github.com/debanjan110d/ai-web-scraper/blob/main/main.py"
click node_scrape "https://github.com/debanjan110d/ai-web-scraper/blob/main/scrape.py"
click node_extract_body "https://github.com/debanjan110d/ai-web-scraper/blob/main/scrape.py"
click node_clean_body "https://github.com/debanjan110d/ai-web-scraper/blob/main/scrape.py"
click node_split_dom "https://github.com/debanjan110d/ai-web-scraper/blob/main/scrape.py"
click node_parser "https://github.com/debanjan110d/ai-web-scraper/blob/main/parse.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_streamlit,node_user,node_browser toneBlue
class node_scrape,node_extract_body,node_clean_body,node_split_dom toneAmber
class node_parser toneMint
class node_website,node_ollama toneIndigo
```

## Components

### User Interface

- [Scraper UI](https://github.com/debanjan110d/ai-web-scraper/blob/main/main.py): `main.py`

### Page Processing

- [Website Scraper](https://github.com/debanjan110d/ai-web-scraper/blob/main/scrape.py): `scrape.py`
- [Body Extraction](https://github.com/debanjan110d/ai-web-scraper/blob/main/scrape.py): `scrape.py`
- [Text Cleaning](https://github.com/debanjan110d/ai-web-scraper/blob/main/scrape.py): `scrape.py`
- [Text Chunking](https://github.com/debanjan110d/ai-web-scraper/blob/main/scrape.py): `scrape.py`

### AI Extraction

- [Ollama Parser](https://github.com/debanjan110d/ai-web-scraper/blob/main/parse.py): `parse.py`

### Other components

- **User**
- **Chrome Browser**
- **Target Website**
- **Ollama Model**

## Connections

- User → Scraper UI: enters URL
- User → Scraper UI: enters description
- Scraper UI → Website Scraper: calls scrape
- Website Scraper → Chrome Browser: drives browser
- Chrome Browser → Target Website: loads page
- Website Scraper → Scraper UI: returns HTML
- Scraper UI → Body Extraction: calls extraction
- Scraper UI → Text Cleaning: calls cleaning
- Scraper UI → Text Chunking: calls chunking
- Ollama Parser → Ollama Model: invokes model