How rashigupta24/retaillake-aws-data-engineering works
RetailLake is an AWS retail data lake that turns raw CSV/JSON files into validated, enriched Silver Parquet datasets, quarantines invalid records, and prepares six Gold analytics datasets for Athena. The documented Glue Workflow runs customer, product, and coupon jobs before orders, then runs payments and returns; the orders job also reads the Silver reference datasets directly. Crawlers and the Data Catalog make Silver available for querying. Job-level processing is supported by sampled source; workflow orchestration and the Gold query flow follow the README, whose implementation details were not sampled.
Main entry points: ordersJob.py Orders job · PaymentsJob.py Payments job · ReturnsJob.py Returns job · CustomerJob.py Customer job
Components 22
Ingestion and validation
- Glue Workflow
- Customer job
glue/CustomerJob.py - Product job
glue/ProductJob.py - Coupon job
glue/CoupansJob.py - Orders job
glue/ordersJob.py - Payments job
glue/PaymentsJob.py - Returns job
glue/ReturnsJob.py
Lake storage
- Raw S3
- Silver Parquet
- Quarantine
Catalog and query
- Glue Crawlers
- Glue Catalog
- Amazon Athena
Gold analytics
- Gold datasets
- Sales analytics SQL
sql/gold_sales.sql - Customer analytics SQL
sql/gold_customer_analytics.sql - Product analytics SQL
sql/gold_product_analytics.sql - Payment analytics SQL
sql/gold_payment_analytics.sql - Return analytics SQL
sql/gold_return_analytics.sql - Coupon analytics SQL
sql/gold_coupon_analytics.sql
Other components
- Raw files
- Analyst
Connections 48
- Raw files → Raw S3: lands
- Glue Workflow → Customer job: dispatches
- Glue Workflow → Product job: dispatches
- Glue Workflow → Coupon job: dispatches
- Glue Workflow → Orders job: dispatches
- Glue Workflow → Payments job: dispatches
- Glue Workflow → Returns job: dispatches
- Raw S3 → Customer job: reads
- Raw S3 → Product job: reads
- Raw S3 → Coupon job: reads
- Raw S3 → Orders job: reads
- Raw S3 → Payments job: reads
- Raw S3 → Returns job: reads
- Customer job → Silver Parquet: writes
- Product job → Silver Parquet: writes
- Coupon job → Silver Parquet: writes
- Orders job → Silver Parquet: writes
- Payments job → Silver Parquet: writes
- Returns job → Silver Parquet: writes
- Customer job → Quarantine: writes
- Product job → Quarantine: writes
- Coupon job → Quarantine: writes
- Orders job → Quarantine: writes
- Payments job → Quarantine: writes
- Returns job → Quarantine: writes
- Silver Parquet → Orders job: lookup reads
- Silver Parquet → Payments job: order lookup
- Silver Parquet → Returns job: order lookup
- Silver Parquet → Glue Crawlers: scans
- Glue Crawlers → Glue Catalog: updates
- Glue Catalog → Amazon Athena: supplies metadata
- Analyst → Amazon Athena: queries
- Amazon Athena → Sales analytics SQL: executes
- Amazon Athena → Customer analytics SQL: executes
- Amazon Athena → Product analytics SQL: executes
- Amazon Athena → Payment analytics SQL: executes
- Amazon Athena → Return analytics SQL: executes
- Amazon Athena → Coupon analytics SQL: executes
- Sales analytics SQL → Silver Parquet: queries
- Customer analytics SQL → Silver Parquet: queries
- Product analytics SQL → Silver Parquet: queries
- Payment analytics SQL → Silver Parquet: queries
- Return analytics SQL → Silver Parquet: queries
- Coupon analytics SQL → Silver Parquet: queries
- Sales analytics SQL → Gold datasets: prepares
- Customer analytics SQL → Gold datasets: prepares
- Product analytics SQL → Gold datasets: prepares
- Payment analytics SQL → Gold datasets: prepares