How nassim-assassi/protein-source-helper works
Protein Source Helper is currently a Morrisons-focused grocery-data scraper and PostgreSQL ingestion repository; the public comparison dashboard and scheduled refresh are planned, not implemented. The runtime workflow is a CLI-triggered retailer ETL: the orchestrator creates a run, dynamically dispatches to retailer extraction and transformation, then loads normalized products and snapshots. PostgreSQL schema and views support the planned catalog and freshness-aware comparisons. The README documents broader dashboard capabilities, but this tree contains no dashboard implementation. Relationships below follow sampled calls or explicit README statements; unsampled subsystems and automation are not inferred.
Main entry points: retailer_etl.py Retailer ETL · load.py Catalog Loader · scrape_run_helpers.py Run Tracking · extract.py Morrisons Extraction
Components 13
ETL Control
- Retailer ETL
scrapers/orchestrators/retailer_etl.py - Run Tracking
scrapers/orchestrators/scrape_run_helpers.py - ETL Contracts
scrapers/etl_contracts.py - DB Configuration
scrapers/config.py - Retailer Identity
scrapers/retailer_config.py
Retailer Adapter
- Morrisons Extraction
scrapers/morrisons_scraper/extract.py
Normalization
- Product Normalization
scrapers/morrisons_scraper/transform.py - Loader Columns
scrapers/constants.py
Catalog Storage
- Catalog Loader
scrapers/loader/load.py - PostgreSQL Catalog
database/schema.sql - Comparison Views
database/views.sql
Other components
- ETL CLI
scrapers/orchestrators/retailer_etl.py - Morrisons Site
Connections 18
- ETL CLI → Retailer ETL: invokes
scrapers/orchestrators/retailer_etl.py - Retailer ETL → Run Tracking: starts or fails run
scrapers/orchestrators/retailer_etl.py - Retailer ETL → Morrisons Extraction: dispatches extraction
scrapers/orchestrators/retailer_etl.py - Morrisons Extraction → Morrisons Site: fetches pages
scrapers/morrisons_scraper/extract.py - Retailer ETL → Product Normalization: dispatches transformation
scrapers/orchestrators/retailer_etl.py - Retailer ETL → Catalog Loader: loads results
scrapers/orchestrators/retailer_etl.py - Retailer ETL → ETL Contracts: uses result types
scrapers/orchestrators/retailer_etl.py - Morrisons Extraction → ETL Contracts: returns extraction result
scrapers/morrisons_scraper/extract.py - Product Normalization → ETL Contracts: returns transform result
scrapers/morrisons_scraper/transform.py - Catalog Loader → Loader Columns: uses product columns
scrapers/loader/load.py - Retailer ETL → DB Configuration: reads settings
scrapers/orchestrators/retailer_etl.py - Retailer ETL → Retailer Identity: creates retailer config
scrapers/orchestrators/retailer_etl.py - Run Tracking → Catalog Loader: resolves retailer
scrapers/orchestrators/scrape_run_helpers.py - Run Tracking → DB Configuration: reads settings
scrapers/orchestrators/scrape_run_helpers.py - Run Tracking → Retailer Identity: uses retailer identity
scrapers/orchestrators/scrape_run_helpers.py - Catalog Loader → DB Configuration: connects using settings
scrapers/loader/load.py - Catalog Loader → PostgreSQL Catalog: upserts catalog and snapshots
scrapers/loader/load.py - Comparison Views → PostgreSQL Catalog: reads catalog tables
README.md