How yuvarajgitcat/scrapeultimate works
Project type and purpose
• ScrapeUltimate is an open-source, Python-based web scraping application.
• Purpose: Provide a universal scraper for static and JavaScript-rendered sites, with AI-driven extraction, attended mode for captchas/logins, pagination/infinite-scroll handling, and flexible output (JSON/CSV/Excel).
File structure
• Top-level scripts:
• scraper.py – core scraping logic (Selenium, BeautifulSoup, html2text)
• pagination_detector.py – logic to detect and iterate pages
• api_management.py & api_test.py – wrapper around AI APIs (OpenAI GPT, Llama)
• streamlit_app.py – Streamlit UI
• requirements.txt – dependencies
• output/… – per-session folders containing rawData, sorted_data in .md/.json/.xlsx
• External binaries: chromedriver-win64 (Selenium driver).
• Virtual env: .venv, __pycache__, .gitignore.
README for architecture hints
• Methodology steps I–XI outline the data pipeline:
1. HTML fetching & preprocessing (Selenium + scroll simulation)
2. Conversion to Markdown
3. AI-based schema generation & extraction
4. Local LLM processing via LLM Studio
5. Dynamic Pydantic models
6. Data cleaning, pagination, infinite scroll, attended mode
7. Output to file system
Main system components & interactions
a. Components:
• User (interacts via CLI or Streamlit UI)
• Streamlit frontend (streamlit_app.py)
• Scraper Engine
• Selenium + ChromeDriver (HTML fetch & dynamic interactions)
• BeautifulSoup + html2text (cleaning & Markdown conversion)
• pagination_detector (detect & loop pages)
• AI Processing Layer
• OpenAI GPT (cloud API calls via api_management)
• Local LLM (via LLM Studio & Llama model)
• Data Modeling
• Pydantic dynamic schemas
• Output Service
• JSON/CSV/Excel writer (assets.py helpers)
• File system storage (output/<session>/…)
• External Services
• OpenAI API endpoint
• Local LLM Studio server
b. Relationships & Data flows:
1. User → Streamlit UI → Scraper Engine (URL + settings)
2. Scraper Engine → Selenium/ChromeDriver → fetch raw HTML
3. Scraper Engine → BeautifulSoup/html2text → cleaned Markdown
4. Markdown → API Management → OpenAI GPT → schema suggestions & extraction
5. Markdown → Local LLM Studio → advanced processing (summarization, classification)
6. Extracted data → Pydantic models → cleaned JSON/Excel
7. Write to file system → user download
c. Architectural patterns & principles:
• Pipeline (fetch → clean → transform → extract → output)
• Modular separation (fetcher, parser, AI client, output writer)
• Extensibility hooks (attended mode, pagination, infinite scroll)
d. Key technologies/frameworks:
• Python, Selenium, ChromeDriver, BeautifulSoup, html2text
• OpenAI GPT API, Llama via LLM Studio
• Pydantic for dynamic schemas
• Streamlit for UI
• pandas/openpyxl for Excel