How techwithtim/ai-web-scraper works
Project Type and Purpose
• This is a standalone, command-line Python tool (“AI-Web-Scraper”) whose job is to scrape web pages via Selenium (using chromedriver) and Brightdata proxies, then post-process the scraped content—optionally calling an AI engine (Ollama) for parsing or enrichment.
File Structure
• Flat structure – no nested packages:
• main.py: the orchestration/CLI entry point
• scrape.py: contains functions–launches Selenium, configures proxies, fetches raw HTML
• parse.py: post-processing routines—cleans up, extracts data, may invoke Ollama
• chromedriver: the browser driver binary
• requirements.txt: pinned Python dependencies (selenium, brightdata SDK, ollama-client, dotenv…)
• sample.env: template for environment variables (proxy credentials, AI-model settings, target URLs)
README
• High-level description: “An AI web scraper using ollama, brightdata, selenium and other libraries.”
1. User / CLI (main.py)
2. Env Loader (.env / sample.env via python-dotenv)
3. Scrape Module (scrape.py)
4. Selenium + ChromeDriver
5. Brightdata Proxy Service (external API)
6. Target Websites (external systems)
7. Parse Module (parse.py)
8. Ollama AI Engine (local server or cloud endpoint)
9. Output Sink (console, file, or downstream DB)
• User invokes main.py → main loads .env → main calls scrape.scrape_pages()
• scrape.py spins Selenium with ChromeDriver, configures proxy via Brightdata SDK → sends HTTP requests to Target Websites → collects raw HTML
• scrape.py returns raw data to main → main invokes parse.parse_content()
• parse.py optionally calls Ollama API for NLP/ML processing → receives structured results
• final data written to Output Sink (e.g., CSV, JSON, database)
c. Architectural Patterns & Principles:
• Single-responsibility modules (Orchestrator vs. Scraper vs. Parser)
• Clear separation of concerns (I/O, scraping, parsing, AI integration)
• Environment-driven configuration (12-factor app)
d. Key Technologies & Libraries:
• Python 3.x
• Selenium WebDriver + ChromeDriver
• Brightdata Proxy SDK
• Ollama Python client or HTTP API
• python-dotenv for env management