How petervlaskovits/grocery-pipeline works
This repository is a Prefect-orchestrated batch ETL pipeline for grocery-store analytics. The main workflow extracts three PostgreSQL tables, applies table-specific Spark cleaning and Pandera validation, then writes Parquet files to S3 for Redshift staging and analytics. The repository also includes SQL for warehouse modeling and KPI queries, plus exploratory-analysis notebooks. Source excerpts show the orchestration and transformation calls; README statements support the broader warehouse and dashboard relationships, whose implementations are not sampled.
Main entry points: main.py Prefect pipeline · inventory.py Inventory transforms · suppliers.py Supplier transforms · transactions.py Transaction transforms
Components 18
Pipeline orchestration
- Prefect pipeline
src/pipeline/main.py - Table extraction
src/pipeline/tasks/extract.py - Parquet S3 loader
src/pipeline/tasks/load.py
Cleaning and validation
- Inventory transforms
src/pipeline/tasks/transform/inventory.py - Supplier transforms
src/pipeline/tasks/transform/suppliers.py - Transaction transforms
src/pipeline/tasks/transform/transactions.py - Shared cleaning maps
src/pipeline/utils/cleaning.py - Pandera schemas
src/pipeline/utils/quality_checks.py
Warehouse analytics
- S3 data lake
- Redshift warehouse
- Warehouse model SQL
src/sql_queries/redshift/data_modelling - KPI query SQL
src/sql_queries/redshift/skus
Data exploration
- Inventory EDA
src/eda_notebooks/inventory_eda.ipynb - Supplier EDA
src/eda_notebooks/suppliers_eda.ipynb - Transaction EDA
src/eda_notebooks/transactions_eda.ipynb
Other components
- Pipeline operator
- PostgreSQL source
- Analytics dashboard
Connections 18
- Pipeline operator → Prefect pipeline: starts
README.md - Prefect pipeline → Table extraction: calls extract
src/pipeline/main.py - Table extraction → PostgreSQL source: reads tables
src/pipeline/tasks/extract.py - Prefect pipeline → Inventory transforms: calls transform
src/pipeline/main.py - Prefect pipeline → Supplier transforms: calls transform
src/pipeline/main.py - Prefect pipeline → Transaction transforms: calls transform
src/pipeline/main.py - Inventory transforms → Shared cleaning maps: uses mappings
src/pipeline/tasks/transform/inventory.py - Supplier transforms → Shared cleaning maps: uses mappings
src/pipeline/tasks/transform/suppliers.py - Transaction transforms → Shared cleaning maps: uses mappings
src/pipeline/tasks/transform/transactions.py - Inventory transforms → Pandera schemas: applies schema
src/pipeline/tasks/transform/inventory.py - Supplier transforms → Pandera schemas: applies schema
src/pipeline/tasks/transform/suppliers.py - Transaction transforms → Pandera schemas: applies schema
src/pipeline/tasks/transform/transactions.py - Prefect pipeline → Parquet S3 loader: calls load
src/pipeline/main.py - Parquet S3 loader → S3 data lake: uploads Parquet
src/pipeline/tasks/load.py - S3 data lake → Redshift warehouse: loads staging data
README.md - Warehouse model SQL → Redshift warehouse: models warehouse
README.md - KPI query SQL → Redshift warehouse: queries KPIs
README.md - Redshift warehouse → Analytics dashboard: powers analytics
README.md