How mhashir03/scrapefy works
Overall project type and purpose.
• Scrapefy is an AI-powered web scraper. Its goal is to automate data extraction from websites, clean the content with HTML parsing, and further process it using a language model (Ollama) based on custom instructions provided by the user.
• The tool uses a Streamlit interface for user interaction, Selenium for automating browser tasks, BeautifulSoup for HTML parsing, and integrates a language model for AI parsing.
File structure.
• The project root contains several Python scripts (main.py, parse.py, scrape.py) which likely separate responsibilities:
• main.py: Likely the entry point that wires up the various components.
• scrape.py: Responsible for controlling Selenium and scraping website data.
• parse.py: Likely handles the HTML parsing (and possibly interaction with the language model) using BeautifulSoup and/or Langchain.
Components and interactions.
• User Interface (UI): The front-end built using Streamlit, where users input URLs and custom parsing instructions.
• Web Scraping Module: Driven by Selenium (in scrape.py) to automate web browser interactions and fetch website content.
• HTML Parsing Module: Uses BeautifulSoup (in parse.py) to clean and extract the relevant HTML content.
• AI Parsing Module: Integrates a large language model (Ollama) to process the extracted content, which could be further supported by other libraries such as Langchain.
• Data flow: The UI collects user input, triggers the scraping module, passes the scraped data to the parsing modules sequentially, and finally displays the AI processed data back on the UI.
Architectural patterns and design principles.
• The design employs a modular approach, separating responsibilities into distinct scripts.
• There is a clear separation between input (UI via Streamlit), processing (Selenium and BeautifulSoup), and output (displaying results on the UI).
• Sequential data flow where each component’s output serves as the input for the next, ensuring clarity in the data-processing pipeline.
Relevant technologies and external dependencies.
• Python forms the backbone of the project.
• Streamlit provides a fast way to create the UI.
• Selenium is used for automating the web scraping process.
• BeautifulSoup handles the extraction and cleaning of the HTML content.
• Ollama (and possibly Langchain) are used for AI-based parsing of data.
• A note on ChromeDriver: The README indicates that a ChromeDriver must be available, so include a component for external driver configuration and integration.