How shihab-miah/daraz-product-scraper-tool works
1. What this repo is
- A single-purpose Python scraper for Daraz Bangladesh product listings.
- Its output is not stored locally; it is pushed to an external Google Sheets/Apps Script web app.
2. Core runtime shape
- The repository is essentially one executable unit: scraper.py.
- requirements.txt defines the Python dependencies needed to run that scraper, including browser automation and HTML parsing libraries.
3. Execution environment
- The tool is designed to run in two contexts:
- GitHub Actions on a schedule, as the primary deployment model.
- Local Python execution for manual runs and debugging.
- The workflow file under .github/workflows/run.yml is the automation entrypoint for hosted runs.
4. Scraping boundary
- The scraper reads product data from Daraz search/result pages.
- The README indicates it uses a combination of Playwright and BeautifulSoup4, which implies a two-step scrape boundary:
- browser-driven page loading for dynamic content,
- HTML parsing for extracting product fields.
5. Data extraction model
- The main data units are product records containing at least:
- title
- price
- discount
- product URL
- one or more image URLs
- The scraper appears to iterate across a configurable set of Daraz search URLs, rather than scraping a single fixed page.
6. Deduplication / “new items only” behavior
- The README says the scraper checks for products already saved and only sends new ones.
- That means the system has a compare-before-send step, likely using previously stored product titles/URLs as the exclusion set.
7. Image validation subsystem
- A notable internal concern is image quality.
- The README calls out a “3-step validation process” for product images, so image URLs are not blindly accepted; they pass additional checks before being included in the outbound record.
8. Outbound integration
- The scraper sends data over HTTP using requests to a Google Apps Script Web App URL supplied through the WEB_APP_URL environment variable.
- This is the main integration boundary: the scraper does not directly write to Google Sheets; it posts JSON to a lightweight receiver.
9. External data sink
- The Google Apps Script endpoint acts as a thin ingestion layer for a Google Sheet with a fixed tab name: Products.
- The sheet is the durable storage layer for the scraped catalog data.
10. Operational flow
- Scheduled GitHub Actions run the Python scraper.
- The scraper loads search pages, extracts candidate products, filters out duplicates, validates images, and POSTs new records to the Apps Script URL.
- The Apps Script app appends the data to the sheet and exposes a GET endpoint for duplicate checking.
11. Configuration and secrets
- The only clearly required secret is WEB_APP_URL, stored in GitHub Actions secrets or as a local environment variable.
- Search scope is configurable in the scraper itself via a URL list mentioned in the README.
12. Important architectural note
- This repo is not a full application stack; it is a scheduled data-collection worker plus an external spreadsheet receiver.
- The diagram should center on the pipeline: GitHub Actions / local Python → Daraz pages → scraper.py → Google Apps Script → Google Sheet.
Main entry points: scraper.py scraper.py · requirements.txt requirements · run.yml run.yml
Components 15
Repository
Core code and automation in this repo.
- run.yml · GitHub Actions entrypoint that triggers scheduled runs.
.github/workflows/run.yml - scraper.py · Main executable that loads pages, extracts products, deduplicates, validates images, and sends new items.
scraper.py - Deduper · Filters out products already stored and keeps only new items.
scraper.py - Image Check · Performs the multi-step image validation before records are accepted.
scraper.py - Product Records · Title, price, discount, URL, and image URLs for each product.
Execution
Ways the scraper is started and run.
- GitHub Actions · Scheduled hosted runtime for the scraper.
.github/workflows/run.yml - Local Python · Local execution for debugging and ad hoc runs.
- WEB_APP_URL · Environment variable pointing to the Apps Script endpoint.
Daraz Source
Product listing pages being scraped.
- Daraz Pages · Daraz Bangladesh product listing and search result pages.
Spreadsheet Sink
External ingestion and durable storage.
- Apps Script · Receives POSTs from the scraper and exposes duplicate-checking GET access.
- Products Sheet · Durable storage tab for scraped catalog data.
Dependencies
Python libraries used by the scraper.
- requirements · Python packages needed for scraping, parsing, and HTTP posting.
requirements.txt - Playwright · Loads dynamic Daraz pages before parsing.
- BeautifulSoup4 · Extracts product fields from rendered HTML.
- requests · Posts scraped product data to the external web app.
Connections 18
- GitHub Actions → run.yml: runs · Hosted scheduled execution starts from the workflow.
- run.yml → scraper.py: invokes · The workflow launches the scraper script.
- Local Python → scraper.py: runs · The same scraper can be executed locally.
- requirements → Playwright: includes · Declares browser automation dependency.
- requirements → BeautifulSoup4: includes · Declares HTML parsing dependency.
- requirements → requests: includes · Declares HTTP client dependency.
- scraper.py → Playwright: loads pages · Uses browser automation to render dynamic product pages.
- Playwright → Daraz Pages: fetches · Retrieves Daraz search and listing pages.
- scraper.py → BeautifulSoup4: parses html · Extracts product fields from rendered page markup.
- scraper.py → Product Records: builds · Converts parsed listings into product records.
- Product Records → Deduper: filters · Candidate records are checked against previously stored items.
- Product Records → Image Check: validates · Image URLs are evaluated before upload.
- Deduper → requests: sends new · Only unseen products proceed to outbound posting.
- Image Check → requests: approved records · Validated records are prepared for transmission.
- WEB_APP_URL → requests: configures · Environment variable supplies the endpoint URL.
- requests → Apps Script: POSTs · Scraper submits new product data to the Apps Script web app.
- Apps Script → Products Sheet: appends · Apps Script writes received records into the Products sheet.
- Apps Script → Deduper: GET lookup · Duplicate checking uses the web app as the source of existing records.