# techwithtim/ai-web-scraper architecture

> Architecture diagram of the GitHub repository techwithtim/ai-web-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/techwithtim/ai-web-scraper
- Repository: https://github.com/techwithtim/ai-web-scraper

## Overview

##### Project Type and Purpose
- This is a standalone, command-line Python tool (“AI-Web-Scraper”) whose job is to scrape web pages via Selenium (using chromedriver) and Brightdata proxies, then post-process the scraped content—optionally calling an AI engine (Ollama) for parsing or enrichment.

##### File Structure
- Flat structure – no nested packages:
- main.py: the orchestration/CLI entry point
- scrape.py: contains functions–launches Selenium, configures proxies, fetches raw HTML
- parse.py: post-processing routines—cleans up, extracts data, may invoke Ollama
- chromedriver: the browser driver binary
- requirements.txt: pinned Python dependencies (selenium, brightdata SDK, ollama-client, dotenv…)
- sample.env: template for environment variables (proxy credentials, AI-model settings, target URLs)

##### README
- High-level description: “An AI web scraper using ollama, brightdata, selenium and other libraries.”

1. User / CLI (main.py)
2. Env Loader (.env / sample.env via python-dotenv)
3. Scrape Module (scrape.py)
4. Selenium + ChromeDriver
5. Brightdata Proxy Service (external API)
6. Target Websites (external systems)
7. Parse Module (parse.py)
8. Ollama AI Engine (local server or cloud endpoint)
9. Output Sink (console, file, or downstream DB)

- User invokes main.py → main loads .env → main calls scrape.scrape_pages()
- scrape.py spins Selenium with ChromeDriver, configures proxy via Brightdata SDK → sends HTTP requests to Target Websites → collects raw HTML
- scrape.py returns raw data to main → main invokes parse.parse_content()
- parse.py optionally calls Ollama API for NLP/ML processing → receives structured results
- final data written to Output Sink (e.g., CSV, JSON, database)

c. Architectural Patterns & Principles:
- Single-responsibility modules (Orchestrator vs. Scraper vs. Parser)
- Clear separation of concerns (I/O, scraping, parsing, AI integration)
- Environment-driven configuration (12-factor app)

d. Key Technologies & Libraries:
- Python 3.x
- Selenium WebDriver + ChromeDriver
- Brightdata Proxy SDK
- Ollama Python client or HTTP API
- python-dotenv for env management

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Local Application Layer
    subgraph "Local Application"
        direction TB
        CLI["User / CLI\n(main.py)"]:::module
        Env["Env Loader\n(sample.env)"]:::module
        Scraper["Scrape Module\n(scrape.py)"]:::module
        Selenium["Selenium\n+ ChromeDriver"]:::module
        Parser["Parse Module\n(parse.py)"]:::module
        Req["Dependencies\n(requirements.txt)"]:::datastore
        Docs["Documentation\n(README.md)"]:::datastore
        Output["Output Sink\n(Console/File/DB)"]:::datastore
    end

    %% External Services Layer
    subgraph "External Services"
        direction TB
        Proxy["Brightdata Proxy Service"]:::external
        Sites["Target Websites"]:::external
        AI["Ollama AI Engine"]:::external
    end

    %% Data Flows
    CLI -->|"load .env"| Env
    CLI -->|"invoke scrape_pages()"| Scraper
    Scraper -->|"launch WebDriver"| Selenium
    Selenium -->|"uses binary"| SeleniumDriver["ChromeDriver"]:::module
    Scraper -->|"configure proxy via SDK"| Proxy
    Selenium -->|"HTTP(S) requests"| Sites
    Scraper -->|"return raw HTML"| CLI
    CLI -->|"invoke parse_content()"| Parser
    Parser -->|"optional NLP/ML"| AI
    Parser -->|"write results"| Output

    %% References
    CLI -.-> Req
    CLI -.-> Docs

    %% Click Events
    click CLI "https://github.com/techwithtim/ai-web-scraper/blob/main/main.py"
    click Env "https://github.com/techwithtim/ai-web-scraper/blob/main/sample.env"
    click Scraper "https://github.com/techwithtim/ai-web-scraper/blob/main/scrape.py"
    click SeleniumDriver "https://github.com/techwithtim/ai-web-scraper/tree/main/chromedriver"
    click Parser "https://github.com/techwithtim/ai-web-scraper/blob/main/parse.py"
    click Req "https://github.com/techwithtim/ai-web-scraper/blob/main/requirements.txt"
    click Docs "https://github.com/techwithtim/ai-web-scraper/blob/main/README.md"

    %% Styles
    classDef module fill:#bbdefb,stroke:#1e88e5,color:#0d47a1;
    classDef external fill:#c8e6c9,stroke:#388e3c,color:#1b5e20;
    classDef datastore fill:#ffe0b2,stroke:#ef6c00,color:#e65100;
```
