How debanjan110d/ai-web-scraper works
Ai-Web-Scraper is a small Streamlit application for fetching a website, cleaning its page text, and preparing it for description-driven extraction. The implemented UI calls the scraping and content-processing functions, while parse.py independently defines Ollama-backed extraction over text chunks. The sampled main.py ends after chunking and does not call that parser or display parsed results, so that connection is not inferred. The README documents a results table and automatic Ollama use, but these are not supported by the sampled implementation.
Main entry points: main.py Scraper UI · scrape.py Website Scraper · parse.py Ollama Parser
Components 10
User Interface
- Scraper UI
main.py
Page Processing
- Website Scraper
scrape.py - Body Extraction
scrape.py - Text Cleaning
scrape.py - Text Chunking
scrape.py
AI Extraction
- Ollama Parser
parse.py
Other components
- User
- Chrome Browser
- Target Website
- Ollama Model
Connections 10
- User → Scraper UI: enters URL
- User → Scraper UI: enters description
- Scraper UI → Website Scraper: calls scrape
- Website Scraper → Chrome Browser: drives browser
- Chrome Browser → Target Website: loads page
- Website Scraper → Scraper UI: returns HTML
- Scraper UI → Body Extraction: calls extraction
- Scraper UI → Text Cleaning: calls cleaning
- Scraper UI → Text Chunking: calls chunking
- Ollama Parser → Ollama Model: invokes model