# dtungpka/shopee-scraper architecture

> Architecture diagram of the GitHub repository dtungpka/shopee-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dtungpka/shopee-scraper
- Repository: https://github.com/dtungpka/shopee-scraper

## Overview

Shopee Scraper has two user-facing workflows: search Shopee for products and collect product details/reviews, then optionally label saved comments for sentiment. The scraper supports index-only retrieval, review limits, and per-star collection; the labeling utility supports manual and OpenAI-based classification. The sampled scraper excerpt omits parts of its orchestration and persistence code, so uncertain internal links are not shown.

Main entry points: [`src/retriv.py`](https://github.com/dtungpka/shopee-scraper/blob/main/src/retriv.py) (Scraper CLI), [`src/json_labeling.py`](https://github.com/dtungpka/shopee-scraper/blob/main/src/json_labeling.py) (Manual labeling)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_scrape["Product scraping"]
  node_scraper["Scraper CLI<br/>[retriv.py]"]
  node_browser["Chrome browser"]
  node_shopee["Shopee"]
  node_search["Product search<br/>[retriv.py]"]
  node_index["Product index<br/>[retriv.py]"]
  node_details["Product details<br/>[retriv.py]"]
  node_reviews["Review collection<br/>[retriv.py]"]
  node_cookies["Cookie file"]
  node_product_json["Product JSON"]
end

subgraph group_label["Sentiment labeling"]
  node_label_cli["Labeling CLI<br/>[json_labeling.py]"]
  node_comment_data["Comment JSON"]
  node_flatten["Comment selection<br/>[json_labeling.py]"]
  node_manual["Manual labeling<br/>[json_labeling.py]"]
  node_automatic["Automatic labeling<br/>[json_labeling.py]"]
  node_clipboard["Clipboard"]
  node_openai["OpenAI API"]
  node_restore["Label restoration<br/>[json_labeling.py]"]
end

node_user(("User"))

node_user -->|"runs"| node_scraper
node_scraper -->|"launches"| node_browser
node_search -->|"navigates"| node_browser
node_browser -->|"requests pages"| node_shopee
node_search -->|"extracts products"| node_index
node_index -->|"enriches products"| node_details
node_details -->|"collects ratings"| node_reviews
node_scraper -->|"loads cookies"| node_cookies
node_scraper -->|"saves results"| node_product_json
node_user -->|"runs"| node_label_cli
node_label_cli -->|"loads and saves"| node_comment_data
node_label_cli -->|"selects comments"| node_flatten
node_flatten -->|"provides chunks"| node_manual
node_flatten -->|"provides chunks"| node_automatic
node_manual -->|"requests labels"| node_user
node_manual -.->|"copies prompt"| node_clipboard
node_automatic -.->|"requests labels"| node_openai
node_manual -->|"returns labels"| node_restore
node_automatic -->|"returns labels"| node_restore
node_restore -->|"writes labels"| node_comment_data

click node_scraper "https://github.com/dtungpka/shopee-scraper/blob/main/src/retriv.py"
click node_search "https://github.com/dtungpka/shopee-scraper/blob/main/src/retriv.py"
click node_index "https://github.com/dtungpka/shopee-scraper/blob/main/src/retriv.py"
click node_details "https://github.com/dtungpka/shopee-scraper/blob/main/src/retriv.py"
click node_reviews "https://github.com/dtungpka/shopee-scraper/blob/main/src/retriv.py"
click node_label_cli "https://github.com/dtungpka/shopee-scraper/blob/main/src/json_labeling.py"
click node_flatten "https://github.com/dtungpka/shopee-scraper/blob/main/src/json_labeling.py"
click node_manual "https://github.com/dtungpka/shopee-scraper/blob/main/src/json_labeling.py"
click node_automatic "https://github.com/dtungpka/shopee-scraper/blob/main/src/json_labeling.py"
click node_restore "https://github.com/dtungpka/shopee-scraper/blob/main/src/json_labeling.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scraper,node_browser,node_shopee,node_search,node_index,node_details,node_reviews,node_cookies,node_product_json,node_user toneBlue
class node_label_cli,node_comment_data,node_flatten,node_manual,node_automatic,node_clipboard,node_openai,node_restore toneAmber
```

## Components

### Product scraping

- [Scraper CLI](https://github.com/dtungpka/shopee-scraper/blob/main/src/retriv.py): `src/retriv.py`
- **Chrome browser**
- **Shopee**
- [Product search](https://github.com/dtungpka/shopee-scraper/blob/main/src/retriv.py): `src/retriv.py`
- [Product index](https://github.com/dtungpka/shopee-scraper/blob/main/src/retriv.py): `src/retriv.py`
- [Product details](https://github.com/dtungpka/shopee-scraper/blob/main/src/retriv.py): `src/retriv.py`
- [Review collection](https://github.com/dtungpka/shopee-scraper/blob/main/src/retriv.py): `src/retriv.py`
- **Cookie file**
- **Product JSON**

### Sentiment labeling

- [Labeling CLI](https://github.com/dtungpka/shopee-scraper/blob/main/src/json_labeling.py): `src/json_labeling.py`
- **Comment JSON**
- [Comment selection](https://github.com/dtungpka/shopee-scraper/blob/main/src/json_labeling.py): `src/json_labeling.py`
- [Manual labeling](https://github.com/dtungpka/shopee-scraper/blob/main/src/json_labeling.py): `src/json_labeling.py`
- [Automatic labeling](https://github.com/dtungpka/shopee-scraper/blob/main/src/json_labeling.py): `src/json_labeling.py`
- **Clipboard**
- **OpenAI API**
- [Label restoration](https://github.com/dtungpka/shopee-scraper/blob/main/src/json_labeling.py): `src/json_labeling.py`

### Other components

- **User**

## Connections

- User → Scraper CLI: runs
- Scraper CLI → Chrome browser: launches
- Product search → Chrome browser: navigates
- Chrome browser → Shopee: requests pages
- Product search → Product index: extracts products
- Product index → Product details: enriches products
- Product details → Review collection: collects ratings
- Scraper CLI → Cookie file: loads cookies
- Scraper CLI → Product JSON: saves results
- User → Labeling CLI: runs
- Labeling CLI → Comment JSON: loads and saves
- Labeling CLI → Comment selection: selects comments
- Comment selection → Manual labeling: provides chunks
- Comment selection → Automatic labeling: provides chunks
- Manual labeling → User: requests labels
- Manual labeling → Clipboard: copies prompt
- Automatic labeling → OpenAI API: requests labels
- Manual labeling → Label restoration: returns labels
- Automatic labeling → Label restoration: returns labels
- Label restoration → Comment JSON: writes labels