# nassim-assassi/protein-source-helper architecture

> Architecture diagram of the GitHub repository nassim-assassi/protein-source-helper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nassim-assassi/protein-source-helper
- Repository: https://github.com/nassim-assassi/protein-source-helper

## Overview

Protein Source Helper is currently a Morrisons-focused grocery-data scraper and PostgreSQL ingestion repository; the public comparison dashboard and scheduled refresh are planned, not implemented. The runtime workflow is a CLI-triggered retailer ETL: the orchestrator creates a run, dynamically dispatches to retailer extraction and transformation, then loads normalized products and snapshots. PostgreSQL schema and views support the planned catalog and freshness-aware comparisons. The README documents broader dashboard capabilities, but this tree contains no dashboard implementation. Relationships below follow sampled calls or explicit README statements; unsampled subsystems and automation are not inferred.

Main entry points: [`scrapers/orchestrators/retailer_etl.py`](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/orchestrators/retailer_etl.py) (Retailer ETL), [`scrapers/loader/load.py`](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/loader/load.py) (Catalog Loader), [`scrapers/orchestrators/scrape_run_helpers.py`](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/orchestrators/scrape_run_helpers.py) (Run Tracking), [`scrapers/morrisons_scraper/extract.py`](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/morrisons_scraper/extract.py) (Morrisons Extraction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["ETL Control"]
  node_orchestrator["Retailer ETL<br/>[retailer_etl.py]"]
  node_runhelpers["Run Tracking"]
  node_contracts["ETL Contracts<br/>[etl_contracts.py]"]
  node_config["DB Configuration<br/>[config.py]"]
  node_retailerconfig["Retailer Identity<br/>[retailer_config.py]"]
end

subgraph group_adapter["Retailer Adapter"]
  node_extractor["Morrisons Extraction<br/>[extract.py]"]
end

subgraph group_normalization["Normalization"]
  node_transformer["Product Normalization<br/>[transform.py]"]
  node_constants["Loader Columns<br/>[constants.py]"]
end

subgraph group_storage["Catalog Storage"]
  node_loader["Catalog Loader<br/>[load.py]"]
  node_postgres[("PostgreSQL Catalog<br/>[schema.sql]")]
  node_comparisonviews[("Comparison Views<br/>[views.sql]")]
end

node_cli(("ETL CLI<br/>[retailer_etl.py]"))
node_retailer(("Morrisons Site"))

node_cli -->|"invokes"| node_orchestrator
node_orchestrator -->|"starts or fails run"| node_runhelpers
node_orchestrator -->|"dispatches extraction"| node_extractor
node_extractor -->|"fetches pages"| node_retailer
node_orchestrator -->|"dispatches transformation"| node_transformer
node_orchestrator -->|"loads results"| node_loader
node_orchestrator -->|"uses result types"| node_contracts
node_extractor -->|"returns extraction result"| node_contracts
node_transformer -->|"returns transform result"| node_contracts
node_loader -->|"uses product columns"| node_constants
node_orchestrator -->|"reads settings"| node_config
node_orchestrator -->|"creates retailer config"| node_retailerconfig
node_runhelpers -->|"resolves retailer"| node_loader
node_runhelpers -->|"reads settings"| node_config
node_runhelpers -->|"uses retailer identity"| node_retailerconfig
node_loader -->|"connects using settings"| node_config
node_loader -->|"upserts catalog and snapshots"| node_postgres
node_comparisonviews -->|"reads catalog tables"| node_postgres

click node_cli "https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/orchestrators/retailer_etl.py"
click node_orchestrator "https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/orchestrators/retailer_etl.py"
click node_runhelpers "https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/orchestrators/scrape_run_helpers.py"
click node_contracts "https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/etl_contracts.py"
click node_config "https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/config.py"
click node_retailerconfig "https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/retailer_config.py"
click node_extractor "https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/morrisons_scraper/extract.py"
click node_transformer "https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/morrisons_scraper/transform.py"
click node_constants "https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/constants.py"
click node_loader "https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/loader/load.py"
click node_postgres "https://github.com/nassim-assassi/protein-source-helper/blob/main/database/schema.sql"
click node_comparisonviews "https://github.com/nassim-assassi/protein-source-helper/blob/main/database/views.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_orchestrator,node_runhelpers,node_contracts,node_config,node_retailerconfig toneBlue
class node_extractor toneAmber
class node_transformer,node_constants toneMint
class node_loader,node_postgres,node_comparisonviews toneRose
class node_cli toneTeal
class node_retailer toneIndigo
```

## Components

### ETL Control

- [Retailer ETL](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/orchestrators/retailer_etl.py): `scrapers/orchestrators/retailer_etl.py`
- [Run Tracking](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/orchestrators/scrape_run_helpers.py): `scrapers/orchestrators/scrape_run_helpers.py`
- [ETL Contracts](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/etl_contracts.py): `scrapers/etl_contracts.py`
- [DB Configuration](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/config.py): `scrapers/config.py`
- [Retailer Identity](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/retailer_config.py): `scrapers/retailer_config.py`

### Retailer Adapter

- [Morrisons Extraction](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/morrisons_scraper/extract.py): `scrapers/morrisons_scraper/extract.py`

### Normalization

- [Product Normalization](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/morrisons_scraper/transform.py): `scrapers/morrisons_scraper/transform.py`
- [Loader Columns](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/constants.py): `scrapers/constants.py`

### Catalog Storage

- [Catalog Loader](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/loader/load.py): `scrapers/loader/load.py`
- [PostgreSQL Catalog](https://github.com/nassim-assassi/protein-source-helper/blob/main/database/schema.sql): `database/schema.sql`
- [Comparison Views](https://github.com/nassim-assassi/protein-source-helper/blob/main/database/views.sql): `database/views.sql`

### Other components

- [ETL CLI](https://github.com/nassim-assassi/protein-source-helper/blob/main/scrapers/orchestrators/retailer_etl.py): `scrapers/orchestrators/retailer_etl.py`
- **Morrisons Site**

## Connections

- ETL CLI → Retailer ETL: invokes (evidence: `scrapers/orchestrators/retailer_etl.py`)
- Retailer ETL → Run Tracking: starts or fails run (evidence: `scrapers/orchestrators/retailer_etl.py`)
- Retailer ETL → Morrisons Extraction: dispatches extraction (evidence: `scrapers/orchestrators/retailer_etl.py`)
- Morrisons Extraction → Morrisons Site: fetches pages (evidence: `scrapers/morrisons_scraper/extract.py`)
- Retailer ETL → Product Normalization: dispatches transformation (evidence: `scrapers/orchestrators/retailer_etl.py`)
- Retailer ETL → Catalog Loader: loads results (evidence: `scrapers/orchestrators/retailer_etl.py`)
- Retailer ETL → ETL Contracts: uses result types (evidence: `scrapers/orchestrators/retailer_etl.py`)
- Morrisons Extraction → ETL Contracts: returns extraction result (evidence: `scrapers/morrisons_scraper/extract.py`)
- Product Normalization → ETL Contracts: returns transform result (evidence: `scrapers/morrisons_scraper/transform.py`)
- Catalog Loader → Loader Columns: uses product columns (evidence: `scrapers/loader/load.py`)
- Retailer ETL → DB Configuration: reads settings (evidence: `scrapers/orchestrators/retailer_etl.py`)
- Retailer ETL → Retailer Identity: creates retailer config (evidence: `scrapers/orchestrators/retailer_etl.py`)
- Run Tracking → Catalog Loader: resolves retailer (evidence: `scrapers/orchestrators/scrape_run_helpers.py`)
- Run Tracking → DB Configuration: reads settings (evidence: `scrapers/orchestrators/scrape_run_helpers.py`)
- Run Tracking → Retailer Identity: uses retailer identity (evidence: `scrapers/orchestrators/scrape_run_helpers.py`)
- Catalog Loader → DB Configuration: connects using settings (evidence: `scrapers/loader/load.py`)
- Catalog Loader → PostgreSQL Catalog: upserts catalog and snapshots (evidence: `scrapers/loader/load.py`)
- Comparison Views → PostgreSQL Catalog: reads catalog tables (evidence: `README.md`)
