How dtungpka/shopee-scraper works
Shopee Scraper has two user-facing workflows: search Shopee for products and collect product details/reviews, then optionally label saved comments for sentiment. The scraper supports index-only retrieval, review limits, and per-star collection; the labeling utility supports manual and OpenAI-based classification. The sampled scraper excerpt omits parts of its orchestration and persistence code, so uncertain internal links are not shown.
Main entry points: retriv.py Scraper CLI · json_labeling.py Manual labeling
Components 18
Product scraping
- Scraper CLI
src/retriv.py - Chrome browser
- Shopee
- Product search
src/retriv.py - Product index
src/retriv.py - Product details
src/retriv.py - Review collection
src/retriv.py - Cookie file
- Product JSON
Sentiment labeling
- Labeling CLI
src/json_labeling.py - Comment JSON
- Comment selection
src/json_labeling.py - Manual labeling
src/json_labeling.py - Automatic labeling
src/json_labeling.py - Clipboard
- OpenAI API
- Label restoration
src/json_labeling.py
Other components
- User
Connections 20
- User → Scraper CLI: runs
- Scraper CLI → Chrome browser: launches
- Product search → Chrome browser: navigates
- Chrome browser → Shopee: requests pages
- Product search → Product index: extracts products
- Product index → Product details: enriches products
- Product details → Review collection: collects ratings
- Scraper CLI → Cookie file: loads cookies
- Scraper CLI → Product JSON: saves results
- User → Labeling CLI: runs
- Labeling CLI → Comment JSON: loads and saves
- Labeling CLI → Comment selection: selects comments
- Comment selection → Manual labeling: provides chunks
- Comment selection → Automatic labeling: provides chunks
- Manual labeling → User: requests labels
- Manual labeling → Clipboard: copies prompt
- Automatic labeling → OpenAI API: requests labels
- Manual labeling → Label restoration: returns labels
- Automatic labeling → Label restoration: returns labels
- Label restoration → Comment JSON: writes labels