# shihab-miah/daraz-product-scraper-tool architecture

> Architecture diagram of the GitHub repository shihab-miah/daraz-product-scraper-tool, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shihab-miah/daraz-product-scraper-tool
- Repository: https://github.com/shihab-miah/daraz-product-scraper-tool

## Overview

1. **What this repo is**
   - A single-purpose Python scraper for **Daraz Bangladesh** product listings.
   - Its output is not stored locally; it is pushed to an external **Google Sheets/Apps Script web app**.

2. **Core runtime shape**
   - The repository is essentially one executable unit: **`scraper.py`**.
   - `requirements.txt` defines the Python dependencies needed to run that scraper, including browser automation and HTML parsing libraries.

3. **Execution environment**
   - The tool is designed to run in two contexts:
     - **GitHub Actions** on a schedule, as the primary deployment model.
     - **Local Python execution** for manual runs and debugging.
   - The workflow file under `.github/workflows/run.yml` is the automation entrypoint for hosted runs.

4. **Scraping boundary**
   - The scraper reads product data from **Daraz search/result pages**.
   - The README indicates it uses a combination of **Playwright** and **BeautifulSoup4**, which implies a two-step scrape boundary:
     - browser-driven page loading for dynamic content,
     - HTML parsing for extracting product fields.

5. **Data extraction model**
   - The main data units are product records containing at least:
     - title
     - price
     - discount
     - product URL
     - one or more image URLs
   - The scraper appears to iterate across a configurable set of Daraz search URLs, rather than scraping a single fixed page.

6. **Deduplication / “new items only” behavior**
   - The README says the scraper checks for products already saved and only sends new ones.
   - That means the system has a compare-before-send step, likely using previously stored product titles/URLs as the exclusion set.

7. **Image validation subsystem**
   - A notable internal concern is image quality.
   - The README calls out a “3-step validation process” for product images, so image URLs are not blindly accepted; they pass additional checks before being included in the outbound record.

8. **Outbound integration**
   - The scraper sends data over HTTP using **requests** to a **Google Apps Script Web App URL** supplied through the `WEB_APP_URL` environment variable.
   - This is the main integration boundary: the scraper does not directly write to Google Sheets; it posts JSON to a lightweight receiver.

9. **External data sink**
   - The Google Apps Script endpoint acts as a thin ingestion layer for a **Google Sheet** with a fixed tab name: **`Products`**.
   - The sheet is the durable storage layer for the scraped catalog data.

10. **Operational flow**
    - Scheduled GitHub Actions run the Python scraper.
    - The scraper loads search pages, extracts candidate products, filters out duplicates, validates images, and POSTs new records to the Apps Script URL.
    - The Apps Script app appends the data to the sheet and exposes a GET endpoint for duplicate checking.

11. **Configuration and secrets**
    - The only clearly required secret is `WEB_APP_URL`, stored in GitHub Actions secrets or as a local environment variable.
    - Search scope is configurable in the scraper itself via a URL list mentioned in the README.

12. **Important architectural note**
    - This repo is not a full application stack; it is a **scheduled data-collection worker** plus an external spreadsheet receiver.
    - The diagram should center on the pipeline: **GitHub Actions / local Python → Daraz pages → scraper.py → Google Apps Script → Google Sheet**.

Main entry points: [`scraper.py`](https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/scraper.py) (scraper.py), [`requirements.txt`](https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/requirements.txt) (requirements), [`.github/workflows/run.yml`](https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/.github/workflows/run.yml) (run.yml)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_repo["Repository"]
  node_workflow["run.yml<br/>workflow<br/>[run.yml]"]
  node_scraper["scraper.py<br/>scraper pipeline<br/>[scraper.py]"]
  node_dedupe{{"Deduper<br/>compare-before-send<br/>[scraper.py]"}}
  node_image_check{{"Image Check<br/>validation<br/>[scraper.py]"}}
  node_payloads["Product Records<br/>structured data"]
end

subgraph group_runtime["Execution"]
  node_github_actions(("GitHub Actions<br/>scheduler<br/>[run.yml]"))
  node_local_python(("Local Python<br/>manual runtime"))
  node_web_app_url["WEB_APP_URL<br/>config secret"]
end

subgraph group_source["Daraz Source"]
  node_daraz_pages[("Daraz Pages<br/>search pages")]
end

subgraph group_sink["Spreadsheet Sink"]
  node_web_app["Apps Script<br/>web app"]
  node_sheet[("Products Sheet<br/>google sheet")]
end

subgraph group_deps["Dependencies"]
  node_requirements["requirements<br/>python deps<br/>[requirements.txt]"]
  node_playwright["Playwright<br/>browser automation"]
  node_beautifulsoup["BeautifulSoup4<br/>html parser"]
  node_requests["requests<br/>http client"]
end

node_github_actions -->|"runs"| node_workflow
node_workflow -->|"invokes"| node_scraper
node_local_python -->|"runs"| node_scraper
node_requirements -->|"includes"| node_playwright
node_requirements -->|"includes"| node_beautifulsoup
node_requirements -->|"includes"| node_requests
node_scraper -->|"loads pages"| node_playwright
node_playwright -->|"fetches"| node_daraz_pages
node_scraper -->|"parses html"| node_beautifulsoup
node_scraper -->|"builds"| node_payloads
node_payloads -->|"filters"| node_dedupe
node_payloads -->|"validates"| node_image_check
node_dedupe -->|"sends new"| node_requests
node_image_check -->|"approved records"| node_requests
node_web_app_url -->|"configures"| node_requests
node_requests -->|"POSTs"| node_web_app
node_web_app -->|"appends"| node_sheet
node_web_app -.->|"GET lookup"| node_dedupe

click node_github_actions "https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/.github/workflows/run.yml"
click node_workflow "https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/.github/workflows/run.yml"
click node_scraper "https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/scraper.py"
click node_requirements "https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/requirements.txt"
click node_dedupe "https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/scraper.py"
click node_image_check "https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/scraper.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_workflow,node_scraper,node_dedupe,node_image_check,node_payloads toneBlue
class node_github_actions,node_local_python,node_web_app_url toneAmber
class node_daraz_pages toneMint
class node_web_app,node_sheet toneRose
class node_requirements,node_playwright,node_beautifulsoup,node_requests toneIndigo
```

## Components

### Repository

Core code and automation in this repo.

- [run.yml](https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/.github/workflows/run.yml): workflow · GitHub Actions entrypoint that triggers scheduled runs. · `.github/workflows/run.yml`
- [scraper.py](https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/scraper.py): scraper pipeline · Main executable that loads pages, extracts products, deduplicates, validates images, and sends new items. · `scraper.py`
- [Deduper](https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/scraper.py): compare-before-send · Filters out products already stored and keeps only new items. · `scraper.py`
- [Image Check](https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/scraper.py): validation · Performs the multi-step image validation before records are accepted. · `scraper.py`
- **Product Records**: structured data · Title, price, discount, URL, and image URLs for each product.

### Execution

Ways the scraper is started and run.

- [GitHub Actions](https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/.github/workflows/run.yml): scheduler · Scheduled hosted runtime for the scraper. · `.github/workflows/run.yml`
- **Local Python**: manual runtime · Local execution for debugging and ad hoc runs.
- **WEB_APP_URL**: config secret · Environment variable pointing to the Apps Script endpoint.

### Daraz Source

Product listing pages being scraped.

- **Daraz Pages**: search pages · Daraz Bangladesh product listing and search result pages.

### Spreadsheet Sink

External ingestion and durable storage.

- **Apps Script**: web app · Receives POSTs from the scraper and exposes duplicate-checking GET access.
- **Products Sheet**: google sheet · Durable storage tab for scraped catalog data.

### Dependencies

Python libraries used by the scraper.

- [requirements](https://github.com/shihab-miah/daraz-product-scraper-tool/blob/main/requirements.txt): python deps · Python packages needed for scraping, parsing, and HTTP posting. · `requirements.txt`
- **Playwright**: browser automation · Loads dynamic Daraz pages before parsing.
- **BeautifulSoup4**: html parser · Extracts product fields from rendered HTML.
- **requests**: http client · Posts scraped product data to the external web app.

## Connections

- GitHub Actions → run.yml: runs · Hosted scheduled execution starts from the workflow.
- run.yml → scraper.py: invokes · The workflow launches the scraper script.
- Local Python → scraper.py: runs · The same scraper can be executed locally.
- requirements → Playwright: includes · Declares browser automation dependency.
- requirements → BeautifulSoup4: includes · Declares HTML parsing dependency.
- requirements → requests: includes · Declares HTTP client dependency.
- scraper.py → Playwright: loads pages · Uses browser automation to render dynamic product pages.
- Playwright → Daraz Pages: fetches · Retrieves Daraz search and listing pages.
- scraper.py → BeautifulSoup4: parses html · Extracts product fields from rendered page markup.
- scraper.py → Product Records: builds · Converts parsed listings into product records.
- Product Records → Deduper: filters · Candidate records are checked against previously stored items.
- Product Records → Image Check: validates · Image URLs are evaluated before upload.
- Deduper → requests: sends new · Only unseen products proceed to outbound posting.
- Image Check → requests: approved records · Validated records are prepared for transmission.
- WEB_APP_URL → requests: configures · Environment variable supplies the endpoint URL.
- requests → Apps Script: POSTs · Scraper submits new product data to the Apps Script web app.
- Apps Script → Products Sheet: appends · Apps Script writes received records into the Products sheet.
- Apps Script → Deduper: GET lookup · Duplicate checking uses the web app as the source of existing records.
