How aswina1636/gmail-data_warehouse-pipeline works
This repository is an email ETL pipeline that turns a Gmail Excel export into a PostgreSQL warehouse for BI. The main workflow is orchestrated by the Prefect flow: it prepares the schema, extracts and loads raw rows, transforms them into deduplicated and classified staging rows, builds marts, and validates row counts. README documents a star-schema mart; its SQL implementation is present in the tree but was not excerpted, so that subsystem is represented without unsupported internal detail.
Main entry points: flow.py Prefect pipeline · 02_build_marts.sql Star-schema marts · load_postgres.py Raw loader · extract_emails.py Excel extraction
Components 14
Orchestration
- Prefect pipeline
flow.py - Runtime config
src/config.py - Run validation
flow.py
Ingestion
- Email export
- Excel extraction
src/extract/extract_emails.py
Warehouse layers
- Raw loader
src/load/load_postgres.py - Raw emails
- Clean and classify
src/transform/clean.py - Staging loader
src/load/load_postgres.py - Staging emails
- Warehouse schema
sql/01_schema.sql
Analytics modeling
- Star-schema marts
sql/02_build_marts.sql
Other components
- Pipeline operator
- BI consumers
Connections 16
- Pipeline operator → Prefect pipeline: starts
README.md - Prefect pipeline → Runtime config: reads settings
flow.py - Prefect pipeline → Warehouse schema: applies DDL
flow.py - Prefect pipeline → Excel extraction: calls extraction
flow.py - Email export → Excel extraction: provides workbook
README.md - Prefect pipeline → Raw loader: loads raw
flow.py - Raw loader → Raw emails: writes rows
src/load/load_postgres.py - Prefect pipeline → Clean and classify: calls transform
flow.py - Prefect pipeline → Staging loader: loads staging
flow.py - Staging loader → Staging emails: writes rows
src/load/load_postgres.py - Prefect pipeline → Star-schema marts: executes mart SQL
flow.py - Prefect pipeline → Run validation: checks counts
flow.py - Run validation → Raw emails: counts rows
flow.py - Run validation → Staging emails: counts rows
flow.py - Run validation → Star-schema marts: checks fact rows
flow.py - BI consumers → Star-schema marts: queries marts
README.md