How subashini-jothilingam/retail-customer-360 works
This repository is a retail Customer 360 analytics platform: source data is ingested and transformed through Bronze, Silver, and Gold processing, then modeled into business analytics. The README describes Airflow orchestration, incremental processing, Snowflake CDC and SCD2 capabilities, and dbt marts. Source excerpts verify several PySpark reads, writes, and validations; unsampled subsystem links are shown only where the README documents them.
Main entry points: ddl Snowflake tables · retail_pipeline.py Airflow pipeline · bronze Bronze ingestion · transform_customers.py Customer cleansing
Components 23
Orchestration and sources
- Retail source data
data/retail360_data - Marketplace demographics
- Airflow pipeline
airflow/dags/retail_pipeline.py - Raw S3 files
PySpark processing
- Bronze ingestion
pyspark_src/bronze - Incremental sales load
pyspark_src/bronze/incremental_sales_load.py - Customer cleansing
pyspark_src/silver/transform_customers.py - Sales cleansing
pyspark_src/silver/transform_sales.py - Store standardization
pyspark_src/silver/transform_stores.py - Customer 360
pyspark_src/gold/customer_360.py - Product performance
pyspark_src/gold/product_performance.py - Store performance
pyspark_src/gold/store_performance.py
Snowflake warehouse
- Snowflake tables
snowflake/ddl - Sales change capture
snowflake/streams/sales_stream.sql - Customer refresh task
snowflake/tasks/load_customer360.sql - Customer SCD2 dimension
snowflake/procedures/sp_load_dim_customer.sql
Analytics modeling
- Staging models
dbt_project/models/staging - Intermediate models
dbt_project/models/intermediate - Business marts
dbt_project/models/marts - BI reporting
Shared processing utilities
- Data quality checks
pyspark_src/common/data_quality.py - Snowflake connection config
pyspark_src/common/snowflake_utils.py - Pipeline logging
pyspark_src/common/logger.py
Connections 28
- Retail source data → Raw S3 files: lands files
README.md - Marketplace demographics → Raw S3 files: lands files
README.md - Airflow pipeline → Bronze ingestion: orchestrates
README.md - Airflow pipeline → Customer cleansing: orchestrates
README.md - Airflow pipeline → Sales cleansing: orchestrates
README.md - Airflow pipeline → Customer 360: orchestrates
README.md - Airflow pipeline → Business marts: runs dbt
README.md - Raw S3 files → Bronze ingestion: reads files
pyspark_src/bronze/load_sales.py - Bronze ingestion → Snowflake tables: writes raw tables
pyspark_src/bronze/load_sales.py - Incremental sales load → Snowflake tables: loads incrementally
README.md - Customer cleansing → Snowflake tables: reads and writes
pyspark_src/silver/transform_customers.py - Sales cleansing → Snowflake tables: reads and writes
pyspark_src/silver/transform_sales.py - Store standardization → Snowflake tables: reads and writes
pyspark_src/silver/transform_stores.py - Customer cleansing → Data quality checks: validates
pyspark_src/silver/transform_customers.py - Sales cleansing → Data quality checks: validates
pyspark_src/silver/transform_sales.py - Bronze ingestion → Snowflake connection config: uses connection options
pyspark_src/bronze/load_sales.py - Customer cleansing → Snowflake connection config: uses connection options
pyspark_src/silver/transform_customers.py - Customer 360 → Snowflake tables: reads and writes
pyspark_src/gold/customer_360.py - Product performance → Snowflake tables: reads and writes
pyspark_src/gold/product_performance.py - Store performance → Snowflake tables: reads and writes
pyspark_src/gold/store_performance.py - Customer 360 → Snowflake connection config: uses connection options
pyspark_src/gold/customer_360.py - Snowflake tables → Staging models: models warehouse data
README.md - Staging models → Intermediate models: builds intermediate models
- Intermediate models → Business marts: builds marts
- Business marts → BI reporting: feeds analytics
README.md - Snowflake tables → Sales change capture: tracks sales changes
README.md - Customer refresh task → Customer SCD2 dimension: refreshes customer data
README.md - Pipeline logging → Bronze ingestion: logs sales load
pyspark_src/bronze/load_sales.py