# allankipchumba/scrape-carrefour architecture

> Architecture diagram of the GitHub repository allankipchumba/scrape-carrefour, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/allankipchumba/scrape-carrefour
- Repository: https://github.com/allankipchumba/scrape-carrefour

## Overview

Carrefour Kenya Product Scraper turns a selected category/subcategory run into brand-level product workbooks, then provides a separate aggregation step for category workbooks and a summary report. The main path is category selection → brand discovery → paginated product extraction → structured parsing → Excel output. The README also documents an optional OpenAI HTML summarizer and reseller-ready aggregate exports. Source excerpts establish the runtime calls and file transformations; aggregation is a separate invocation, not called by the scraper entry point.

Main entry points: [`main.py`](https://github.com/allankipchumba/scrape-carrefour/blob/main/main.py) (Run coordinator), [`aggregate_categories.py`](https://github.com/allankipchumba/scrape-carrefour/blob/main/aggregate_categories.py) (Category aggregation), [`scraper/scraper.py`](https://github.com/allankipchumba/scrape-carrefour/blob/main/scraper/scraper.py) (Product pagination), [`aggregated_data`](https://github.com/allankipchumba/scrape-carrefour/tree/main/aggregated_data) (Category workbooks)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_scrape["Catalogue scraping"]
  node_main["Run coordinator<br/>[main.py]"]
  node_driver["Browser setup<br/>[driver.py]"]
  node_brands["Brand discovery<br/>[scraper.py]"]
  node_products["Product pagination<br/>[scraper.py]"]
  node_parser["Product extraction<br/>[scraper.py]"]
end

subgraph group_outputs["Workbook outputs"]
  node_brandfiles["Brand workbooks"]
  node_writer["Excel writer<br/>[writer.py]"]
  node_aggregator["Category aggregation"]
  node_categoryfiles["Category workbooks"]
  node_summary["Summary report"]
end

subgraph group_insights["Optional insights"]
  node_summarizer["HTML summarizer<br/>[summarizer.py]"]
end

node_operator(("Scraper operator"))
node_carrefour["Carrefour website"]
node_reseller(("Reseller"))
node_openai["OpenAI API"]

node_operator -->|"starts run"| node_main
node_main -->|"creates browser"| node_driver
node_main -->|"discovers brands"| node_brands
node_main -->|"scrapes brands"| node_products
node_main -->|"writes products"| node_writer
node_brands -->|"loads categories"| node_carrefour
node_products -->|"loads products"| node_carrefour
node_products -->|"parses rows"| node_parser
node_writer -->|"writes workbooks"| node_brandfiles
node_aggregator -->|"reads workbooks"| node_brandfiles
node_aggregator -->|"writes aggregates"| node_categoryfiles
node_aggregator -->|"generates report"| node_summary
node_summary -->|"writes report"| node_categoryfiles
node_reseller -->|"imports feed"| node_categoryfiles
node_summarizer -->|"requests summary"| node_openai

click node_main "https://github.com/allankipchumba/scrape-carrefour/blob/main/main.py"
click node_driver "https://github.com/allankipchumba/scrape-carrefour/blob/main/scraper/driver.py"
click node_brands "https://github.com/allankipchumba/scrape-carrefour/blob/main/scraper/scraper.py"
click node_products "https://github.com/allankipchumba/scrape-carrefour/blob/main/scraper/scraper.py"
click node_parser "https://github.com/allankipchumba/scrape-carrefour/blob/main/scraper/scraper.py"
click node_brandfiles "https://github.com/allankipchumba/scrape-carrefour/tree/main/data"
click node_writer "https://github.com/allankipchumba/scrape-carrefour/blob/main/scraper/writer.py"
click node_aggregator "https://github.com/allankipchumba/scrape-carrefour/blob/main/aggregate_categories.py"
click node_categoryfiles "https://github.com/allankipchumba/scrape-carrefour/tree/main/aggregated_data"
click node_summary "https://github.com/allankipchumba/scrape-carrefour/blob/main/aggregate_categories.py"
click node_summarizer "https://github.com/allankipchumba/scrape-carrefour/blob/main/insights/summarizer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_driver,node_brands,node_products,node_parser toneBlue
class node_brandfiles,node_writer,node_aggregator,node_categoryfiles,node_summary toneAmber
class node_summarizer,node_openai toneMint
class node_operator,node_carrefour,node_reseller toneIndigo
```

## Components

### Catalogue scraping

- [Run coordinator](https://github.com/allankipchumba/scrape-carrefour/blob/main/main.py): `main.py`
- [Browser setup](https://github.com/allankipchumba/scrape-carrefour/blob/main/scraper/driver.py): `scraper/driver.py`
- [Brand discovery](https://github.com/allankipchumba/scrape-carrefour/blob/main/scraper/scraper.py): `scraper/scraper.py`
- [Product pagination](https://github.com/allankipchumba/scrape-carrefour/blob/main/scraper/scraper.py): `scraper/scraper.py`
- [Product extraction](https://github.com/allankipchumba/scrape-carrefour/blob/main/scraper/scraper.py): `scraper/scraper.py`

### Workbook outputs

- [Brand workbooks](https://github.com/allankipchumba/scrape-carrefour/tree/main/data): `data`
- [Excel writer](https://github.com/allankipchumba/scrape-carrefour/blob/main/scraper/writer.py): `scraper/writer.py`
- [Category aggregation](https://github.com/allankipchumba/scrape-carrefour/blob/main/aggregate_categories.py): `aggregate_categories.py`
- [Category workbooks](https://github.com/allankipchumba/scrape-carrefour/tree/main/aggregated_data): `aggregated_data`
- [Summary report](https://github.com/allankipchumba/scrape-carrefour/blob/main/aggregate_categories.py): `aggregate_categories.py`

### Optional insights

- [HTML summarizer](https://github.com/allankipchumba/scrape-carrefour/blob/main/insights/summarizer.py): `insights/summarizer.py`

### Other components

- **Scraper operator**
- **Carrefour website**
- **Reseller**
- **OpenAI API**

## Connections

- Scraper operator → Run coordinator: starts run (evidence: `README.md`)
- Run coordinator → Browser setup: creates browser (evidence: `main.py`)
- Run coordinator → Brand discovery: discovers brands (evidence: `main.py`)
- Run coordinator → Product pagination: scrapes brands (evidence: `main.py`)
- Run coordinator → Excel writer: writes products (evidence: `main.py`)
- Brand discovery → Carrefour website: loads categories (evidence: `scraper/scraper.py`)
- Product pagination → Carrefour website: loads products (evidence: `scraper/scraper.py`)
- Product pagination → Product extraction: parses rows (evidence: `scraper/scraper.py`)
- Excel writer → Brand workbooks: writes workbooks (evidence: `scraper/writer.py`)
- Category aggregation → Brand workbooks: reads workbooks (evidence: `aggregate_categories.py`)
- Category aggregation → Category workbooks: writes aggregates (evidence: `aggregate_categories.py`)
- Category aggregation → Summary report: generates report (evidence: `aggregate_categories.py`)
- Summary report → Category workbooks: writes report (evidence: `aggregate_categories.py`)
- Reseller → Category workbooks: imports feed (evidence: `README.md`)
- HTML summarizer → OpenAI API: requests summary (evidence: `insights/summarizer.py`)
