# petervlaskovits/grocery-pipeline architecture

> Architecture diagram of the GitHub repository petervlaskovits/grocery-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/petervlaskovits/grocery-pipeline
- Repository: https://github.com/petervlaskovits/grocery-pipeline

## Overview

This repository is a Prefect-orchestrated batch ETL pipeline for grocery-store analytics. The main workflow extracts three PostgreSQL tables, applies table-specific Spark cleaning and Pandera validation, then writes Parquet files to S3 for Redshift staging and analytics. The repository also includes SQL for warehouse modeling and KPI queries, plus exploratory-analysis notebooks. Source excerpts show the orchestration and transformation calls; README statements support the broader warehouse and dashboard relationships, whose implementations are not sampled.

Main entry points: [`src/pipeline/main.py`](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/main.py) (Prefect pipeline), [`src/pipeline/tasks/transform/inventory.py`](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/tasks/transform/inventory.py) (Inventory transforms), [`src/pipeline/tasks/transform/suppliers.py`](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/tasks/transform/suppliers.py) (Supplier transforms), [`src/pipeline/tasks/transform/transactions.py`](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/tasks/transform/transactions.py) (Transaction transforms)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Pipeline orchestration"]
  node_flow["Prefect pipeline<br/>[main.py]"]
  node_extract["Table extraction<br/>[extract.py]"]
  node_loader["Parquet S3 loader<br/>[load.py]"]
end

subgraph group_transforms["Cleaning and validation"]
  node_inventory["Inventory transforms<br/>[inventory.py]"]
  node_suppliers["Supplier transforms<br/>[suppliers.py]"]
  node_transactions["Transaction transforms<br/>[transactions.py]"]
  node_cleaning["Shared cleaning maps<br/>[cleaning.py]"]
  node_validation["Pandera schemas<br/>[quality_checks.py]"]
end

subgraph group_warehouse["Warehouse analytics"]
  node_s3[("S3 data lake")]
  node_redshift[("Redshift warehouse")]
  node_model_sql["Warehouse model SQL"]
  node_kpi_sql["KPI query SQL"]
end

subgraph group_exploration["Data exploration"]
  node_inventory_eda["Inventory EDA"]
  node_supplier_eda["Supplier EDA"]
  node_transaction_eda["Transaction EDA"]
end

node_operator(("Pipeline operator"))
node_postgres[("PostgreSQL source")]
node_dashboard["Analytics dashboard"]

node_operator -->|"starts"| node_flow
node_flow -->|"calls extract"| node_extract
node_extract -->|"reads tables"| node_postgres
node_flow -->|"calls transform"| node_inventory
node_flow -->|"calls transform"| node_suppliers
node_flow -->|"calls transform"| node_transactions
node_inventory -->|"uses mappings"| node_cleaning
node_suppliers -->|"uses mappings"| node_cleaning
node_transactions -->|"uses mappings"| node_cleaning
node_inventory -->|"applies schema"| node_validation
node_suppliers -->|"applies schema"| node_validation
node_transactions -->|"applies schema"| node_validation
node_flow -->|"calls load"| node_loader
node_loader -->|"uploads Parquet"| node_s3
node_s3 -->|"loads staging data"| node_redshift
node_model_sql -->|"models warehouse"| node_redshift
node_kpi_sql -->|"queries KPIs"| node_redshift
node_redshift -->|"powers analytics"| node_dashboard

click node_flow "https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/main.py"
click node_extract "https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/tasks/extract.py"
click node_inventory "https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/tasks/transform/inventory.py"
click node_suppliers "https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/tasks/transform/suppliers.py"
click node_transactions "https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/tasks/transform/transactions.py"
click node_cleaning "https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/utils/cleaning.py"
click node_validation "https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/utils/quality_checks.py"
click node_loader "https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/tasks/load.py"
click node_model_sql "https://github.com/petervlaskovits/grocery-pipeline/tree/master/src/sql_queries/redshift/data_modelling"
click node_kpi_sql "https://github.com/petervlaskovits/grocery-pipeline/tree/master/src/sql_queries/redshift/skus"
click node_inventory_eda "https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/eda_notebooks/inventory_eda.ipynb"
click node_supplier_eda "https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/eda_notebooks/suppliers_eda.ipynb"
click node_transaction_eda "https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/eda_notebooks/transactions_eda.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_flow,node_extract,node_loader toneBlue
class node_inventory,node_suppliers,node_transactions,node_cleaning,node_validation,node_postgres toneAmber
class node_s3,node_redshift,node_model_sql,node_kpi_sql toneMint
class node_inventory_eda,node_supplier_eda,node_transaction_eda toneRose
class node_operator,node_dashboard toneIndigo
```

## Components

### Pipeline orchestration

- [Prefect pipeline](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/main.py): `src/pipeline/main.py`
- [Table extraction](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/tasks/extract.py): `src/pipeline/tasks/extract.py`
- [Parquet S3 loader](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/tasks/load.py): `src/pipeline/tasks/load.py`

### Cleaning and validation

- [Inventory transforms](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/tasks/transform/inventory.py): `src/pipeline/tasks/transform/inventory.py`
- [Supplier transforms](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/tasks/transform/suppliers.py): `src/pipeline/tasks/transform/suppliers.py`
- [Transaction transforms](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/tasks/transform/transactions.py): `src/pipeline/tasks/transform/transactions.py`
- [Shared cleaning maps](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/utils/cleaning.py): `src/pipeline/utils/cleaning.py`
- [Pandera schemas](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/pipeline/utils/quality_checks.py): `src/pipeline/utils/quality_checks.py`

### Warehouse analytics

- **S3 data lake**
- **Redshift warehouse**
- [Warehouse model SQL](https://github.com/petervlaskovits/grocery-pipeline/tree/master/src/sql_queries/redshift/data_modelling): `src/sql_queries/redshift/data_modelling`
- [KPI query SQL](https://github.com/petervlaskovits/grocery-pipeline/tree/master/src/sql_queries/redshift/skus): `src/sql_queries/redshift/skus`

### Data exploration

- [Inventory EDA](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/eda_notebooks/inventory_eda.ipynb): `src/eda_notebooks/inventory_eda.ipynb`
- [Supplier EDA](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/eda_notebooks/suppliers_eda.ipynb): `src/eda_notebooks/suppliers_eda.ipynb`
- [Transaction EDA](https://github.com/petervlaskovits/grocery-pipeline/blob/master/src/eda_notebooks/transactions_eda.ipynb): `src/eda_notebooks/transactions_eda.ipynb`

### Other components

- **Pipeline operator**
- **PostgreSQL source**
- **Analytics dashboard**

## Connections

- Pipeline operator → Prefect pipeline: starts (evidence: `README.md`)
- Prefect pipeline → Table extraction: calls extract (evidence: `src/pipeline/main.py`)
- Table extraction → PostgreSQL source: reads tables (evidence: `src/pipeline/tasks/extract.py`)
- Prefect pipeline → Inventory transforms: calls transform (evidence: `src/pipeline/main.py`)
- Prefect pipeline → Supplier transforms: calls transform (evidence: `src/pipeline/main.py`)
- Prefect pipeline → Transaction transforms: calls transform (evidence: `src/pipeline/main.py`)
- Inventory transforms → Shared cleaning maps: uses mappings (evidence: `src/pipeline/tasks/transform/inventory.py`)
- Supplier transforms → Shared cleaning maps: uses mappings (evidence: `src/pipeline/tasks/transform/suppliers.py`)
- Transaction transforms → Shared cleaning maps: uses mappings (evidence: `src/pipeline/tasks/transform/transactions.py`)
- Inventory transforms → Pandera schemas: applies schema (evidence: `src/pipeline/tasks/transform/inventory.py`)
- Supplier transforms → Pandera schemas: applies schema (evidence: `src/pipeline/tasks/transform/suppliers.py`)
- Transaction transforms → Pandera schemas: applies schema (evidence: `src/pipeline/tasks/transform/transactions.py`)
- Prefect pipeline → Parquet S3 loader: calls load (evidence: `src/pipeline/main.py`)
- Parquet S3 loader → S3 data lake: uploads Parquet (evidence: `src/pipeline/tasks/load.py`)
- S3 data lake → Redshift warehouse: loads staging data (evidence: `README.md`)
- Warehouse model SQL → Redshift warehouse: models warehouse (evidence: `README.md`)
- KPI query SQL → Redshift warehouse: queries KPIs (evidence: `README.md`)
- Redshift warehouse → Analytics dashboard: powers analytics (evidence: `README.md`)
