# subashini-jothilingam/retail-customer-360 architecture

> Architecture diagram of the GitHub repository subashini-jothilingam/retail-customer-360, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/subashini-jothilingam/retail-customer-360
- Repository: https://github.com/subashini-jothilingam/retail-customer-360

## Overview

This repository is a retail Customer 360 analytics platform: source data is ingested and transformed through Bronze, Silver, and Gold processing, then modeled into business analytics. The README describes Airflow orchestration, incremental processing, Snowflake CDC and SCD2 capabilities, and dbt marts. Source excerpts verify several PySpark reads, writes, and validations; unsampled subsystem links are shown only where the README documents them.

Main entry points: [`snowflake/ddl`](https://github.com/subashini-jothilingam/retail-customer-360/tree/main/snowflake/ddl) (Snowflake tables), [`airflow/dags/retail_pipeline.py`](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/airflow/dags/retail_pipeline.py) (Airflow pipeline), [`pyspark_src/bronze`](https://github.com/subashini-jothilingam/retail-customer-360/tree/main/pyspark_src/bronze) (Bronze ingestion), [`pyspark_src/silver/transform_customers.py`](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/silver/transform_customers.py) (Customer cleansing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration and sources"]
  node_retail["Retail source data"]
  node_marketplace["Marketplace demographics"]
  node_airflow["Airflow pipeline<br/>[retail_pipeline.py]"]
  node_s3[("Raw S3 files")]
end

subgraph group_processing["PySpark processing"]
  node_bronze["Bronze ingestion"]
  node_incremental["Incremental sales load"]
  node_silver_customers["Customer cleansing"]
  node_silver_sales["Sales cleansing<br/>[transform_sales.py]"]
  node_silver_stores["Store standardization"]
  node_customer360["Customer 360<br/>[customer_360.py]"]
  node_productperf["Product performance"]
  node_storeperf["Store performance"]
end

subgraph group_warehouse["Snowflake warehouse"]
  node_snowflake[("Snowflake tables")]
  node_cdc["Sales change capture<br/>[sales_stream.sql]"]
  node_tasks["Customer refresh task"]
  node_scd["Customer SCD2 dimension"]
end

subgraph group_modeling["Analytics modeling"]
  node_dbt_staging["Staging models"]
  node_dbt_intermediate["Intermediate models"]
  node_dbt_marts["Business marts"]
  node_reporting(("BI reporting"))
end

subgraph group_quality["Shared processing utilities"]
  node_quality["Data quality checks<br/>[data_quality.py]"]
  node_snowflake_config["Snowflake connection config<br/>[snowflake_utils.py]"]
  node_logger["Pipeline logging<br/>[logger.py]"]
end

node_retail -->|"lands files"| node_s3
node_marketplace -->|"lands files"| node_s3
node_airflow -->|"orchestrates"| node_bronze
node_airflow -->|"orchestrates"| node_silver_customers
node_airflow -->|"orchestrates"| node_silver_sales
node_airflow -->|"orchestrates"| node_customer360
node_airflow -->|"runs dbt"| node_dbt_marts
node_s3 -->|"reads files"| node_bronze
node_bronze -->|"writes raw tables"| node_snowflake
node_incremental -.->|"loads incrementally"| node_snowflake
node_silver_customers -->|"reads and writes"| node_snowflake
node_silver_sales -->|"reads and writes"| node_snowflake
node_silver_stores -->|"reads and writes"| node_snowflake
node_silver_customers -->|"validates"| node_quality
node_silver_sales -->|"validates"| node_quality
node_bronze -->|"uses connection options"| node_snowflake_config
node_silver_customers -->|"uses connection options"| node_snowflake_config
node_customer360 -->|"reads and writes"| node_snowflake
node_productperf -->|"reads and writes"| node_snowflake
node_storeperf -->|"reads and writes"| node_snowflake
node_customer360 -->|"uses connection options"| node_snowflake_config
node_snowflake -->|"models warehouse data"| node_dbt_staging
node_dbt_staging -.->|"builds intermediate models"| node_dbt_intermediate
node_dbt_intermediate -.->|"builds marts"| node_dbt_marts
node_dbt_marts -->|"feeds analytics"| node_reporting
node_snowflake -->|"tracks sales changes"| node_cdc
node_tasks -.->|"refreshes customer data"| node_scd
node_logger -->|"logs sales load"| node_bronze

click node_retail "https://github.com/subashini-jothilingam/retail-customer-360/tree/main/data/retail360_data"
click node_airflow "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/airflow/dags/retail_pipeline.py"
click node_bronze "https://github.com/subashini-jothilingam/retail-customer-360/tree/main/pyspark_src/bronze"
click node_incremental "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/bronze/incremental_sales_load.py"
click node_silver_customers "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/silver/transform_customers.py"
click node_silver_sales "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/silver/transform_sales.py"
click node_silver_stores "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/silver/transform_stores.py"
click node_customer360 "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/gold/customer_360.py"
click node_productperf "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/gold/product_performance.py"
click node_storeperf "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/gold/store_performance.py"
click node_snowflake "https://github.com/subashini-jothilingam/retail-customer-360/tree/main/snowflake/ddl"
click node_cdc "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/snowflake/streams/sales_stream.sql"
click node_tasks "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/snowflake/tasks/load_customer360.sql"
click node_scd "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/snowflake/procedures/sp_load_dim_customer.sql"
click node_dbt_staging "https://github.com/subashini-jothilingam/retail-customer-360/tree/main/dbt_project/models/staging"
click node_dbt_intermediate "https://github.com/subashini-jothilingam/retail-customer-360/tree/main/dbt_project/models/intermediate"
click node_dbt_marts "https://github.com/subashini-jothilingam/retail-customer-360/tree/main/dbt_project/models/marts"
click node_quality "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/common/data_quality.py"
click node_snowflake_config "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/common/snowflake_utils.py"
click node_logger "https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/common/logger.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_retail,node_marketplace,node_airflow,node_s3 toneBlue
class node_bronze,node_incremental,node_silver_customers,node_silver_sales,node_silver_stores,node_customer360,node_productperf,node_storeperf toneAmber
class node_snowflake,node_cdc,node_tasks,node_scd toneMint
class node_dbt_staging,node_dbt_intermediate,node_dbt_marts,node_reporting toneRose
class node_quality,node_snowflake_config,node_logger toneIndigo
```

## Components

### Orchestration and sources

- [Retail source data](https://github.com/subashini-jothilingam/retail-customer-360/tree/main/data/retail360_data): `data/retail360_data`
- **Marketplace demographics**
- [Airflow pipeline](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/airflow/dags/retail_pipeline.py): `airflow/dags/retail_pipeline.py`
- **Raw S3 files**

### PySpark processing

- [Bronze ingestion](https://github.com/subashini-jothilingam/retail-customer-360/tree/main/pyspark_src/bronze): `pyspark_src/bronze`
- [Incremental sales load](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/bronze/incremental_sales_load.py): `pyspark_src/bronze/incremental_sales_load.py`
- [Customer cleansing](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/silver/transform_customers.py): `pyspark_src/silver/transform_customers.py`
- [Sales cleansing](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/silver/transform_sales.py): `pyspark_src/silver/transform_sales.py`
- [Store standardization](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/silver/transform_stores.py): `pyspark_src/silver/transform_stores.py`
- [Customer 360](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/gold/customer_360.py): `pyspark_src/gold/customer_360.py`
- [Product performance](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/gold/product_performance.py): `pyspark_src/gold/product_performance.py`
- [Store performance](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/gold/store_performance.py): `pyspark_src/gold/store_performance.py`

### Snowflake warehouse

- [Snowflake tables](https://github.com/subashini-jothilingam/retail-customer-360/tree/main/snowflake/ddl): `snowflake/ddl`
- [Sales change capture](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/snowflake/streams/sales_stream.sql): `snowflake/streams/sales_stream.sql`
- [Customer refresh task](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/snowflake/tasks/load_customer360.sql): `snowflake/tasks/load_customer360.sql`
- [Customer SCD2 dimension](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/snowflake/procedures/sp_load_dim_customer.sql): `snowflake/procedures/sp_load_dim_customer.sql`

### Analytics modeling

- [Staging models](https://github.com/subashini-jothilingam/retail-customer-360/tree/main/dbt_project/models/staging): `dbt_project/models/staging`
- [Intermediate models](https://github.com/subashini-jothilingam/retail-customer-360/tree/main/dbt_project/models/intermediate): `dbt_project/models/intermediate`
- [Business marts](https://github.com/subashini-jothilingam/retail-customer-360/tree/main/dbt_project/models/marts): `dbt_project/models/marts`
- **BI reporting**

### Shared processing utilities

- [Data quality checks](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/common/data_quality.py): `pyspark_src/common/data_quality.py`
- [Snowflake connection config](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/common/snowflake_utils.py): `pyspark_src/common/snowflake_utils.py`
- [Pipeline logging](https://github.com/subashini-jothilingam/retail-customer-360/blob/main/pyspark_src/common/logger.py): `pyspark_src/common/logger.py`

## Connections

- Retail source data → Raw S3 files: lands files (evidence: `README.md`)
- Marketplace demographics → Raw S3 files: lands files (evidence: `README.md`)
- Airflow pipeline → Bronze ingestion: orchestrates (evidence: `README.md`)
- Airflow pipeline → Customer cleansing: orchestrates (evidence: `README.md`)
- Airflow pipeline → Sales cleansing: orchestrates (evidence: `README.md`)
- Airflow pipeline → Customer 360: orchestrates (evidence: `README.md`)
- Airflow pipeline → Business marts: runs dbt (evidence: `README.md`)
- Raw S3 files → Bronze ingestion: reads files (evidence: `pyspark_src/bronze/load_sales.py`)
- Bronze ingestion → Snowflake tables: writes raw tables (evidence: `pyspark_src/bronze/load_sales.py`)
- Incremental sales load → Snowflake tables: loads incrementally (evidence: `README.md`)
- Customer cleansing → Snowflake tables: reads and writes (evidence: `pyspark_src/silver/transform_customers.py`)
- Sales cleansing → Snowflake tables: reads and writes (evidence: `pyspark_src/silver/transform_sales.py`)
- Store standardization → Snowflake tables: reads and writes (evidence: `pyspark_src/silver/transform_stores.py`)
- Customer cleansing → Data quality checks: validates (evidence: `pyspark_src/silver/transform_customers.py`)
- Sales cleansing → Data quality checks: validates (evidence: `pyspark_src/silver/transform_sales.py`)
- Bronze ingestion → Snowflake connection config: uses connection options (evidence: `pyspark_src/bronze/load_sales.py`)
- Customer cleansing → Snowflake connection config: uses connection options (evidence: `pyspark_src/silver/transform_customers.py`)
- Customer 360 → Snowflake tables: reads and writes (evidence: `pyspark_src/gold/customer_360.py`)
- Product performance → Snowflake tables: reads and writes (evidence: `pyspark_src/gold/product_performance.py`)
- Store performance → Snowflake tables: reads and writes (evidence: `pyspark_src/gold/store_performance.py`)
- Customer 360 → Snowflake connection config: uses connection options (evidence: `pyspark_src/gold/customer_360.py`)
- Snowflake tables → Staging models: models warehouse data (evidence: `README.md`)
- Staging models → Intermediate models: builds intermediate models
- Intermediate models → Business marts: builds marts
- Business marts → BI reporting: feeds analytics (evidence: `README.md`)
- Snowflake tables → Sales change capture: tracks sales changes (evidence: `README.md`)
- Customer refresh task → Customer SCD2 dimension: refreshes customer data (evidence: `README.md`)
- Pipeline logging → Bronze ingestion: logs sales load (evidence: `pyspark_src/bronze/load_sales.py`)
