How parasu-bw/scraping-model works
Scraping-Model collects product-page data through two paths: it can fetch JavaScript-rendered pages into HTML and batch-parse them, or scrape URLs directly in a Playwright session. Both paths extract product name, price, and description and write timestamped JSON results. The batch path separates acquisition, HTML parsing, and product extraction. The README gives no further capability detail; the direct scraper and batch processor are evidenced by source excerpts.
Main entry points: main_processor.py Batch processor · fetcher.py HTML fetcher · playwright_scraper.py Direct scraper · parser.py HTML parser
Components 11
Inputs
- URL list
Model1/prod_url.csv
Page Acquisition
- HTML fetcher
Model1/fetcher.py - Saved HTML
Model1/html_files - Direct scraper
Model1/playwright_scraper.py
Extraction
- HTML parser
Model1/parser.py - Product extractor
Model1/smart_extractor.py - Batch processor
Model1/main_processor.py
Results
- Batch JSON
Model1/scraped_product_data_2025-09-11_15-38-50.json - Direct JSON
Model1/playwright_scraped_data_2025-09-11_14-54-32.json
Other components
- User
- Product websites
Connections 15
- User → HTML fetcher: provides CSV path
Model1/fetcher.py - HTML fetcher → URL list: reads URLs
Model1/fetcher.py - HTML fetcher → Product websites: fetches pages
Model1/fetcher.py - HTML fetcher → Saved HTML: saves HTML
Model1/fetcher.py - User → Direct scraper: provides CSV path
Model1/playwright_scraper.py - Direct scraper → URL list: reads URLs
Model1/playwright_scraper.py - Direct scraper → Product websites: visits pages
Model1/playwright_scraper.py - Direct scraper → Direct JSON: writes JSON
Model1/playwright_scraper.py - User → Batch processor: runs processing
Model1/main_processor.py - Batch processor → Saved HTML: reads files
Model1/main_processor.py - Batch processor → HTML parser: calls parser
Model1/main_processor.py - HTML parser → Batch processor: returns soup
Model1/main_processor.py - Batch processor → Product extractor: calls extractor
Model1/main_processor.py - Product extractor → Batch processor: returns product data
Model1/main_processor.py - Batch processor → Batch JSON: writes JSON
Model1/main_processor.py