# sushil-rgb/daraz-global-webscraper architecture

> Architecture diagram of the GitHub repository sushil-rgb/daraz-global-webscraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sushil-rgb/daraz-global-webscraper
- Repository: https://github.com/sushil-rgb/daraz-global-webscraper

## Overview

Daraz Scraper takes a configured category URL, validates it, scrapes category listings across pages and exports the resulting product records to MongoDB or Excel. The documented product-detail lookup is also represented. The excerpts show the runtime entry point, scraper, and utility calls; selector and user-agent files are present in the tree but their contents were not sampled.

Main entry points: [`main.py`](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/main.py) (Scrape runner), [`scrapers/daraz_scraper.py`](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/daraz_scraper.py) (Daraz scraper), [`scrapers/selectors.yaml`](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/selectors.yaml) (CSS selectors), [`tools/functionalities.py`](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/tools/functionalities.py) (Scrape helpers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Application entry"]
  node_main["Scrape runner<br/>[main.py]"]
end

subgraph group_scrape["Daraz scraping"]
  node_daraz["Daraz scraper<br/>[daraz_scraper.py]"]
  node_category["Category details<br/>[daraz_scraper.py]"]
  node_listing["Category listings<br/>[daraz_scraper.py]"]
  node_product["Product details<br/>[daraz_scraper.py]"]
end

subgraph group_support["Scraping support"]
  node_validate["URL validation<br/>[functionalities.py]"]
  node_domain["Country detection<br/>[functionalities.py]"]
  node_selectors["CSS selectors<br/>[selectors.yaml]"]
  node_helpers["Scrape helpers<br/>[functionalities.py]"]
end

subgraph group_export["Data export"]
  node_mongo["MongoDB export<br/>[daraz_scraper.py]"]
  node_mongodb[("MongoDB")]
  node_excel["Excel export<br/>[daraz_scraper.py]"]
  node_workbook["Excel workbook"]
end

node_user(("Operator"))

node_user -->|"runs"| node_main
node_main -->|"validates URL"| node_validate
node_main -->|"creates scraper"| node_daraz
node_main -->|"selects export"| node_mongo
node_main -->|"selects export"| node_excel
node_daraz -->|"loads helpers"| node_helpers
node_daraz -->|"loads selectors"| node_selectors
node_listing -->|"gets category"| node_category
node_listing -->|"detects country"| node_domain
node_product -->|"queries selectors"| node_selectors
node_product -->|"reads elements"| node_helpers
node_mongo -->|"names collection"| node_category
node_mongo -->|"scrapes records"| node_listing
node_mongo -->|"inserts records"| node_mongodb
node_excel -->|"names workbook"| node_category
node_excel -->|"scrapes records"| node_listing
node_excel -->|"writes workbook"| node_workbook
node_daraz -.->|"offers lookup"| node_product

click node_main "https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/main.py"
click node_validate "https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/tools/functionalities.py"
click node_daraz "https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/daraz_scraper.py"
click node_category "https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/daraz_scraper.py"
click node_listing "https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/daraz_scraper.py"
click node_product "https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/daraz_scraper.py"
click node_domain "https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/tools/functionalities.py"
click node_selectors "https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/selectors.yaml"
click node_helpers "https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/tools/functionalities.py"
click node_mongo "https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/daraz_scraper.py"
click node_excel "https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/daraz_scraper.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main toneBlue
class node_daraz,node_category,node_listing,node_product toneAmber
class node_validate,node_domain,node_selectors,node_helpers toneMint
class node_mongo,node_mongodb,node_excel,node_workbook toneRose
class node_user toneIndigo
```

## Components

### Application entry

- [Scrape runner](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/main.py): `main.py`

### Daraz scraping

- [Daraz scraper](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/daraz_scraper.py): `scrapers/daraz_scraper.py`
- [Category details](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/daraz_scraper.py): `scrapers/daraz_scraper.py`
- [Category listings](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/daraz_scraper.py): `scrapers/daraz_scraper.py`
- [Product details](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/daraz_scraper.py): `scrapers/daraz_scraper.py`

### Scraping support

- [URL validation](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/tools/functionalities.py): `tools/functionalities.py`
- [Country detection](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/tools/functionalities.py): `tools/functionalities.py`
- [CSS selectors](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/selectors.yaml): `scrapers/selectors.yaml`
- [Scrape helpers](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/tools/functionalities.py): `tools/functionalities.py`

### Data export

- [MongoDB export](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/daraz_scraper.py): `scrapers/daraz_scraper.py`
- **MongoDB**
- [Excel export](https://github.com/sushil-rgb/daraz-global-webscraper/blob/master/scrapers/daraz_scraper.py): `scrapers/daraz_scraper.py`
- **Excel workbook**

### Other components

- **Operator**

## Connections

- Operator → Scrape runner: runs (evidence: `README.md`)
- Scrape runner → URL validation: validates URL (evidence: `main.py`)
- Scrape runner → Daraz scraper: creates scraper (evidence: `main.py`)
- Scrape runner → MongoDB export: selects export (evidence: `main.py`)
- Scrape runner → Excel export: selects export (evidence: `main.py`)
- Daraz scraper → Scrape helpers: loads helpers (evidence: `scrapers/daraz_scraper.py`)
- Daraz scraper → CSS selectors: loads selectors (evidence: `scrapers/daraz_scraper.py`)
- Category listings → Category details: gets category (evidence: `scrapers/daraz_scraper.py`)
- Category listings → Country detection: detects country (evidence: `scrapers/daraz_scraper.py`)
- Product details → CSS selectors: queries selectors (evidence: `scrapers/daraz_scraper.py`)
- Product details → Scrape helpers: reads elements (evidence: `scrapers/daraz_scraper.py`)
- MongoDB export → Category details: names collection (evidence: `scrapers/daraz_scraper.py`)
- MongoDB export → Category listings: scrapes records (evidence: `scrapers/daraz_scraper.py`)
- MongoDB export → MongoDB: inserts records (evidence: `scrapers/daraz_scraper.py`)
- Excel export → Category details: names workbook (evidence: `scrapers/daraz_scraper.py`)
- Excel export → Category listings: scrapes records (evidence: `scrapers/daraz_scraper.py`)
- Excel export → Excel workbook: writes workbook (evidence: `scrapers/daraz_scraper.py`)
- Daraz scraper → Product details: offers lookup (evidence: `README.md`)
