How allankipchumba/scrape-carrefour works
Carrefour Kenya Product Scraper turns a selected category/subcategory run into brand-level product workbooks, then provides a separate aggregation step for category workbooks and a summary report. The main path is category selection → brand discovery → paginated product extraction → structured parsing → Excel output. The README also documents an optional OpenAI HTML summarizer and reseller-ready aggregate exports. Source excerpts establish the runtime calls and file transformations; aggregation is a separate invocation, not called by the scraper entry point.
Main entry points: main.py Run coordinator · aggregate_categories.py Category aggregation · scraper.py Product pagination · aggregated_data Category workbooks
Components 15
Catalogue scraping
- Run coordinator
main.py - Browser setup
scraper/driver.py - Brand discovery
scraper/scraper.py - Product pagination
scraper/scraper.py - Product extraction
scraper/scraper.py
Workbook outputs
- Brand workbooks
data - Excel writer
scraper/writer.py - Category aggregation
aggregate_categories.py - Category workbooks
aggregated_data - Summary report
aggregate_categories.py
Optional insights
- HTML summarizer
insights/summarizer.py
Other components
- Scraper operator
- Carrefour website
- Reseller
- OpenAI API
Connections 15
- Scraper operator → Run coordinator: starts run
README.md - Run coordinator → Browser setup: creates browser
main.py - Run coordinator → Brand discovery: discovers brands
main.py - Run coordinator → Product pagination: scrapes brands
main.py - Run coordinator → Excel writer: writes products
main.py - Brand discovery → Carrefour website: loads categories
scraper/scraper.py - Product pagination → Carrefour website: loads products
scraper/scraper.py - Product pagination → Product extraction: parses rows
scraper/scraper.py - Excel writer → Brand workbooks: writes workbooks
scraper/writer.py - Category aggregation → Brand workbooks: reads workbooks
aggregate_categories.py - Category aggregation → Category workbooks: writes aggregates
aggregate_categories.py - Category aggregation → Summary report: generates report
aggregate_categories.py - Summary report → Category workbooks: writes report
aggregate_categories.py - Reseller → Category workbooks: imports feed
README.md - HTML summarizer → OpenAI API: requests summary
insights/summarizer.py