# mhashir03/scrapefy architecture

> Architecture diagram of the GitHub repository mhashir03/scrapefy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mhashir03/scrapefy
- Repository: https://github.com/mhashir03/scrapefy

## Overview

##### Overall project type and purpose.
- Scrapefy is an AI-powered web scraper. Its goal is to automate data extraction from websites, clean the content with HTML parsing, and further process it using a language model (Ollama) based on custom instructions provided by the user.
- The tool uses a Streamlit interface for user interaction, Selenium for automating browser tasks, BeautifulSoup for HTML parsing, and integrates a language model for AI parsing.

##### File structure.
- The project root contains several Python scripts (main.py, parse.py, scrape.py) which likely separate responsibilities:
- main.py: Likely the entry point that wires up the various components.
- scrape.py: Responsible for controlling Selenium and scraping website data.
- parse.py: Likely handles the HTML parsing (and possibly interaction with the language model) using BeautifulSoup and/or Langchain.

##### Components and interactions.
- User Interface (UI): The front-end built using Streamlit, where users input URLs and custom parsing instructions.
- Web Scraping Module: Driven by Selenium (in scrape.py) to automate web browser interactions and fetch website content.
- HTML Parsing Module: Uses BeautifulSoup (in parse.py) to clean and extract the relevant HTML content.
- AI Parsing Module: Integrates a large language model (Ollama) to process the extracted content, which could be further supported by other libraries such as Langchain.
- Data flow: The UI collects user input, triggers the scraping module, passes the scraped data to the parsing modules sequentially, and finally displays the AI processed data back on the UI.

##### Architectural patterns and design principles.
- The design employs a modular approach, separating responsibilities into distinct scripts.
- There is a clear separation between input (UI via Streamlit), processing (Selenium and BeautifulSoup), and output (displaying results on the UI).
- Sequential data flow where each component’s output serves as the input for the next, ensuring clarity in the data-processing pipeline.

##### Relevant technologies and external dependencies.
- Python forms the backbone of the project.
- Streamlit provides a fast way to create the UI.
- Selenium is used for automating the web scraping process.
- BeautifulSoup handles the extraction and cleaning of the HTML content.
- Ollama (and possibly Langchain) are used for AI-based parsing of data.
- A note on ChromeDriver: The README indicates that a ChromeDriver must be available, so include a component for external driver configuration and integration.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "User Interface Layer"
        UI["Streamlit-Based User Interface"]:::ui
    end

    subgraph "Processing Modules"
        Scraper["Web Scraping Module (Selenium)"]:::scraper
        HTML["HTML Parsing Module (BeautifulSoup)"]:::parser
        AI["AI Parsing Module (Ollama/Langchain)"]:::ai
    end

    subgraph "External Dependencies"
        External["External Dependencies (ChromeDriver, Libraries)"]:::external
    end

    UI -->|"UserInput"| Scraper
    Scraper -->|"ScrapedHTML"| HTML
    HTML -->|"CleanedContent"| AI
    AI -->|"ProcessedData"| UI
    External -->|"ProvidesChromeDriver"| Scraper

    click UI "https://github.com/mhashir03/scrapefy/blob/main/main.py"
    click Scraper "https://github.com/mhashir03/scrapefy/blob/main/scrape.py"
    click HTML "https://github.com/mhashir03/scrapefy/blob/main/parse.py"
    click AI "https://github.com/mhashir03/scrapefy/blob/main/parse.py"
    click External "https://github.com/mhashir03/scrapefy/blob/main/req.txt"
    click External "https://github.com/mhashir03/scrapefy/blob/main/README.md"

    classDef ui fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef scraper fill:#dff0d8,stroke:#3c763d,stroke-width:2px;
    classDef parser fill:#fcf8e3,stroke:#8a6d3b,stroke-width:2px;
    classDef ai fill:#f2dede,stroke:#a94442,stroke-width:2px;
    classDef external fill:#d9edf7,stroke:#31708f,stroke-width:2px;
```
