# rashigupta24/retaillake-aws-data-engineering architecture

> Architecture diagram of the GitHub repository rashigupta24/retaillake-aws-data-engineering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rashigupta24/retaillake-aws-data-engineering
- Repository: https://github.com/rashigupta24/retaillake-aws-data-engineering

## Overview

RetailLake is an AWS retail data lake that turns raw CSV/JSON files into validated, enriched Silver Parquet datasets, quarantines invalid records, and prepares six Gold analytics datasets for Athena. The documented Glue Workflow runs customer, product, and coupon jobs before orders, then runs payments and returns; the orders job also reads the Silver reference datasets directly. Crawlers and the Data Catalog make Silver available for querying. Job-level processing is supported by sampled source; workflow orchestration and the Gold query flow follow the README, whose implementation details were not sampled.

Main entry points: [`glue/ordersJob.py`](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/ordersJob.py) (Orders job), [`glue/PaymentsJob.py`](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/PaymentsJob.py) (Payments job), [`glue/ReturnsJob.py`](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/ReturnsJob.py) (Returns job), [`glue/CustomerJob.py`](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/CustomerJob.py) (Customer job)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion and validation"]
  node_workflow["Glue Workflow"]
  node_customer["Customer job<br/>[CustomerJob.py]"]
  node_product["Product job<br/>[ProductJob.py]"]
  node_coupon["Coupon job<br/>[CoupansJob.py]"]
  node_orders["Orders job<br/>[ordersJob.py]"]
  node_payments["Payments job<br/>[PaymentsJob.py]"]
  node_returns["Returns job<br/>[ReturnsJob.py]"]
end

subgraph group_lake["Lake storage"]
  node_raw[("Raw S3")]
  node_silver[("Silver Parquet")]
  node_quarantine[("Quarantine")]
end

subgraph group_catalog["Catalog and query"]
  node_crawlers["Glue Crawlers"]
  node_catalogue[("Glue Catalog")]
  node_athena["Amazon Athena"]
end

subgraph group_analytics["Gold analytics"]
  node_gold[("Gold datasets")]
  node_sales["Sales analytics SQL<br/>[gold_sales.sql]"]
  node_customeranalytics["Customer analytics SQL"]
  node_productanalytics["Product analytics SQL"]
  node_paymentanalytics["Payment analytics SQL"]
  node_returnanalytics["Return analytics SQL"]
  node_couponanalytics["Coupon analytics SQL"]
end

node_source(("Raw files"))
node_analyst(("Analyst"))

node_source -->|"lands"| node_raw
node_workflow -->|"dispatches"| node_customer
node_workflow -->|"dispatches"| node_product
node_workflow -->|"dispatches"| node_coupon
node_workflow -->|"dispatches"| node_orders
node_workflow -->|"dispatches"| node_payments
node_workflow -->|"dispatches"| node_returns
node_raw -->|"reads"| node_customer
node_raw -->|"reads"| node_product
node_raw -->|"reads"| node_coupon
node_raw -->|"reads"| node_orders
node_raw -->|"reads"| node_payments
node_raw -->|"reads"| node_returns
node_customer -->|"writes"| node_silver
node_product -->|"writes"| node_silver
node_coupon -->|"writes"| node_silver
node_orders -->|"writes"| node_silver
node_payments -->|"writes"| node_silver
node_returns -->|"writes"| node_silver
node_customer -->|"writes"| node_quarantine
node_product -->|"writes"| node_quarantine
node_coupon -->|"writes"| node_quarantine
node_orders -->|"writes"| node_quarantine
node_payments -->|"writes"| node_quarantine
node_returns -->|"writes"| node_quarantine
node_silver -->|"lookup reads"| node_orders
node_silver -->|"order lookup"| node_payments
node_silver -->|"order lookup"| node_returns
node_silver -->|"scans"| node_crawlers
node_crawlers -->|"updates"| node_catalogue
node_catalogue -->|"supplies metadata"| node_athena
node_analyst -->|"queries"| node_athena
node_athena -->|"executes"| node_sales
node_athena -->|"executes"| node_customeranalytics
node_athena -->|"executes"| node_productanalytics
node_athena -->|"executes"| node_paymentanalytics
node_athena -->|"executes"| node_returnanalytics
node_athena -->|"executes"| node_couponanalytics
node_sales -->|"queries"| node_silver
node_customeranalytics -->|"queries"| node_silver
node_productanalytics -->|"queries"| node_silver
node_paymentanalytics -->|"queries"| node_silver
node_returnanalytics -->|"queries"| node_silver
node_couponanalytics -->|"queries"| node_silver
node_sales -->|"prepares"| node_gold
node_customeranalytics -->|"prepares"| node_gold
node_productanalytics -->|"prepares"| node_gold
node_paymentanalytics -->|"prepares"| node_gold

click node_customer "https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/CustomerJob.py"
click node_product "https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/ProductJob.py"
click node_coupon "https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/CoupansJob.py"
click node_orders "https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/ordersJob.py"
click node_payments "https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/PaymentsJob.py"
click node_returns "https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/ReturnsJob.py"
click node_sales "https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/sql/gold_sales.sql"
click node_customeranalytics "https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/sql/gold_customer_analytics.sql"
click node_productanalytics "https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/sql/gold_product_analytics.sql"
click node_paymentanalytics "https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/sql/gold_payment_analytics.sql"
click node_returnanalytics "https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/sql/gold_return_analytics.sql"
click node_couponanalytics "https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/sql/gold_coupon_analytics.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_workflow,node_customer,node_product,node_coupon,node_orders,node_payments,node_returns toneBlue
class node_raw,node_silver,node_quarantine toneAmber
class node_crawlers,node_catalogue,node_athena toneMint
class node_gold,node_sales,node_customeranalytics,node_productanalytics,node_paymentanalytics,node_returnanalytics,node_couponanalytics toneRose
class node_source,node_analyst toneIndigo
```

## Components

### Ingestion and validation

- **Glue Workflow**
- [Customer job](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/CustomerJob.py): `glue/CustomerJob.py`
- [Product job](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/ProductJob.py): `glue/ProductJob.py`
- [Coupon job](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/CoupansJob.py): `glue/CoupansJob.py`
- [Orders job](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/ordersJob.py): `glue/ordersJob.py`
- [Payments job](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/PaymentsJob.py): `glue/PaymentsJob.py`
- [Returns job](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/glue/ReturnsJob.py): `glue/ReturnsJob.py`

### Lake storage

- **Raw S3**
- **Silver Parquet**
- **Quarantine**

### Catalog and query

- **Glue Crawlers**
- **Glue Catalog**
- **Amazon Athena**

### Gold analytics

- **Gold datasets**
- [Sales analytics SQL](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/sql/gold_sales.sql): `sql/gold_sales.sql`
- [Customer analytics SQL](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/sql/gold_customer_analytics.sql): `sql/gold_customer_analytics.sql`
- [Product analytics SQL](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/sql/gold_product_analytics.sql): `sql/gold_product_analytics.sql`
- [Payment analytics SQL](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/sql/gold_payment_analytics.sql): `sql/gold_payment_analytics.sql`
- [Return analytics SQL](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/sql/gold_return_analytics.sql): `sql/gold_return_analytics.sql`
- [Coupon analytics SQL](https://github.com/rashigupta24/retaillake-aws-data-engineering/blob/main/sql/gold_coupon_analytics.sql): `sql/gold_coupon_analytics.sql`

### Other components

- **Raw files**
- **Analyst**

## Connections

- Raw files → Raw S3: lands
- Glue Workflow → Customer job: dispatches
- Glue Workflow → Product job: dispatches
- Glue Workflow → Coupon job: dispatches
- Glue Workflow → Orders job: dispatches
- Glue Workflow → Payments job: dispatches
- Glue Workflow → Returns job: dispatches
- Raw S3 → Customer job: reads
- Raw S3 → Product job: reads
- Raw S3 → Coupon job: reads
- Raw S3 → Orders job: reads
- Raw S3 → Payments job: reads
- Raw S3 → Returns job: reads
- Customer job → Silver Parquet: writes
- Product job → Silver Parquet: writes
- Coupon job → Silver Parquet: writes
- Orders job → Silver Parquet: writes
- Payments job → Silver Parquet: writes
- Returns job → Silver Parquet: writes
- Customer job → Quarantine: writes
- Product job → Quarantine: writes
- Coupon job → Quarantine: writes
- Orders job → Quarantine: writes
- Payments job → Quarantine: writes
- Returns job → Quarantine: writes
- Silver Parquet → Orders job: lookup reads
- Silver Parquet → Payments job: order lookup
- Silver Parquet → Returns job: order lookup
- Silver Parquet → Glue Crawlers: scans
- Glue Crawlers → Glue Catalog: updates
- Glue Catalog → Amazon Athena: supplies metadata
- Analyst → Amazon Athena: queries
- Amazon Athena → Sales analytics SQL: executes
- Amazon Athena → Customer analytics SQL: executes
- Amazon Athena → Product analytics SQL: executes
- Amazon Athena → Payment analytics SQL: executes
- Amazon Athena → Return analytics SQL: executes
- Amazon Athena → Coupon analytics SQL: executes
- Sales analytics SQL → Silver Parquet: queries
- Customer analytics SQL → Silver Parquet: queries
- Product analytics SQL → Silver Parquet: queries
- Payment analytics SQL → Silver Parquet: queries
- Return analytics SQL → Silver Parquet: queries
- Coupon analytics SQL → Silver Parquet: queries
- Sales analytics SQL → Gold datasets: prepares
- Customer analytics SQL → Gold datasets: prepares
- Product analytics SQL → Gold datasets: prepares
- Payment analytics SQL → Gold datasets: prepares