# hemug1430/retail_sales_analytics architecture

> Architecture diagram of the GitHub repository hemug1430/retail_sales_analytics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hemug1430/retail_sales_analytics
- Repository: https://github.com/hemug1430/retail_sales_analytics

## Overview

This is an enterprise retail analytics data platform: operators run a Python pipeline that ingests and validates retail files, standardizes and loads them into a MySQL warehouse, reconciles the results, and serves analytics through Power BI. The graph preserves the documented SCD, incremental-load, analytical, and reporting capabilities. Some SQL and dashboard implementation paths are documented in the README but absent from the supplied tree; their internal connections are therefore limited to documented relationships.

Main entry points: [`src/05_16_load_cleaned_data_to_staging.py`](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05_16_load_cleaned_data_to_staging.py) (Staging Load), [`src/data_ingestion.py`](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/data_ingestion.py) (File Ingestion), [`src/config.py`](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/config.py) (Runtime Configuration), [`src/04.13_create_cleaned_staging_datasets.py`](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/04.13_create_cleaned_staging_datasets.py) (Data Standardization)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Intake and Validation"]
  node_ingestion["File Ingestion<br/>[data_ingestion.py]"]
  node_dedupe["Duplicate File Detection"]
  node_schema["Schema Validation"]
  node_quarantine["Invalid File Quarantine"]
  node_audit["Intake Audit Logging"]
end

subgraph group_clean["Standardization and Staging"]
  node_standardize["Data Standardization"]
  node_cleanrules["Cleaning Rules"]
  node_stage["Staging Load"]
  node_stagecheck["Staging Validation"]
end

subgraph group_warehouse["Warehouse Loading"]
  node_dimmerge["Dimension Merges"]
  node_salesload["Sales Incremental Load"]
  node_returnsload["Returns Incremental Load"]
  node_targetload["Targets Incremental Load"]
  node_scd["SCD History Updates"]
  node_mysql[("MySQL Warehouse")]
  node_sql["MySQL Procedures"]
end

subgraph group_analytics["Analytics and Reporting"]
  node_views["Analytical Views"]
  node_date["Date Dimension"]
  node_bi["Power BI Reports"]
end

subgraph group_ops["Reconciliation and Operations"]
  node_e2e["Pipeline Runner"]
  node_config["Runtime Configuration<br/>[config.py]"]
  node_reconcile["Source Reconciliation"]
  node_integrity["Database Integrity"]
  node_backup["Database Backup"]
end

node_operator(("Pipeline Operator"))

node_operator -->|"runs"| node_e2e
node_e2e -.->|"orchestrates"| node_ingestion
node_ingestion -->|"checks hashes"| node_dedupe
node_ingestion -->|"validates schema"| node_schema
node_schema -->|"routes failures"| node_quarantine
node_ingestion -.->|"logs intake"| node_audit
node_ingestion -->|"provides files"| node_standardize
node_standardize -->|"applies cleaning"| node_cleanrules
node_standardize -->|"loads cleaned data"| node_stage
node_stage -.->|"validates counts"| node_stagecheck
node_stage -->|"feeds staging"| node_dimmerge
node_stage -->|"feeds staging"| node_salesload
node_stage -->|"feeds staging"| node_returnsload
node_stage -->|"feeds staging"| node_targetload
node_dimmerge -->|"merges dimensions"| node_mysql
node_salesload -->|"upserts sales"| node_mysql
node_returnsload -->|"loads returns"| node_mysql
node_targetload -->|"loads targets"| node_mysql
node_scd -->|"updates history"| node_mysql
node_mysql -->|"exposes views"| node_views
node_mysql -.->|"supports time analysis"| node_date
node_views -->|"feeds reporting"| node_bi
node_mysql -->|"compares records"| node_reconcile
node_mysql -->|"checks integrity"| node_integrity
node_mysql -->|"backs up data"| node_backup
node_config -->|"configures paths"| node_ingestion
node_config -->|"configures database"| node_stagecheck
node_config -->|"configures database"| node_returnsload

click node_e2e "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/10.01_run_final_e2e_execution.py"
click node_config "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/config.py"
click node_ingestion "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/data_ingestion.py"
click node_dedupe "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/duplicate_file_detection.py"
click node_schema "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/schema_validation.py"
click node_quarantine "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/quarantine_invalid_files.py"
click node_audit "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/intake_audit_log.py"
click node_standardize "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/04.13_create_cleaned_staging_datasets.py"
click node_cleanrules "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/04.11_row_level_business_validation.py"
click node_stage "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05_16_load_cleaned_data_to_staging.py"
click node_stagecheck "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.17_validate_staging_row_counts.py"
click node_dimmerge "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.18_run_staging_to_core_customer_merge.py"
click node_salesload "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.23_run_sales_incremental_insert.py"
click node_returnsload "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.24_run_returns_incremental_insert.py"
click node_targetload "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.25_run_targets_incremental_load.py"
click node_scd "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.27_implement_scd_history_updates.py"
click node_views "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.30_create_core_data_views.py"
click node_date "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/06.01_create_date_dimension.py"
click node_bi "https://github.com/hemug1430/retail_sales_analytics/tree/main/dashboards"
click node_reconcile "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.31_run_source_to_database_reconciliation.py"
click node_integrity "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.33_validate_database_integrity.py"
click node_backup "https://github.com/hemug1430/retail_sales_analytics/blob/main/src/10.05_backup_final_mysql_database.py"
click node_sql "https://github.com/hemug1430/retail_sales_analytics/tree/main/sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingestion,node_dedupe,node_schema,node_quarantine,node_audit toneBlue
class node_standardize,node_cleanrules,node_stage,node_stagecheck toneAmber
class node_dimmerge,node_salesload,node_returnsload,node_targetload,node_scd,node_mysql,node_sql toneMint
class node_views,node_date,node_bi toneRose
class node_e2e,node_config,node_reconcile,node_integrity,node_backup,node_operator toneIndigo
```

## Components

### Intake and Validation

- [File Ingestion](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/data_ingestion.py): `src/data_ingestion.py`
- [Duplicate File Detection](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/duplicate_file_detection.py): `src/duplicate_file_detection.py`
- [Schema Validation](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/schema_validation.py): `src/schema_validation.py`
- [Invalid File Quarantine](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/quarantine_invalid_files.py): `src/quarantine_invalid_files.py`
- [Intake Audit Logging](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/intake_audit_log.py): `src/intake_audit_log.py`

### Standardization and Staging

- [Data Standardization](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/04.13_create_cleaned_staging_datasets.py): `src/04.13_create_cleaned_staging_datasets.py`
- [Cleaning Rules](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/04.11_row_level_business_validation.py): `src/04.11_row_level_business_validation.py`
- [Staging Load](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05_16_load_cleaned_data_to_staging.py): `src/05_16_load_cleaned_data_to_staging.py`
- [Staging Validation](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.17_validate_staging_row_counts.py): `src/05.17_validate_staging_row_counts.py`

### Warehouse Loading

- [Dimension Merges](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.18_run_staging_to_core_customer_merge.py): `src/05.18_run_staging_to_core_customer_merge.py`
- [Sales Incremental Load](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.23_run_sales_incremental_insert.py): `src/05.23_run_sales_incremental_insert.py`
- [Returns Incremental Load](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.24_run_returns_incremental_insert.py): `src/05.24_run_returns_incremental_insert.py`
- [Targets Incremental Load](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.25_run_targets_incremental_load.py): `src/05.25_run_targets_incremental_load.py`
- [SCD History Updates](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.27_implement_scd_history_updates.py): `src/05.27_implement_scd_history_updates.py`
- **MySQL Warehouse**
- [MySQL Procedures](https://github.com/hemug1430/retail_sales_analytics/tree/main/sql): `sql`

### Analytics and Reporting

- [Analytical Views](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.30_create_core_data_views.py): `src/05.30_create_core_data_views.py`
- [Date Dimension](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/06.01_create_date_dimension.py): `src/06.01_create_date_dimension.py`
- [Power BI Reports](https://github.com/hemug1430/retail_sales_analytics/tree/main/dashboards): `dashboards`

### Reconciliation and Operations

- [Pipeline Runner](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/10.01_run_final_e2e_execution.py): `src/10.01_run_final_e2e_execution.py`
- [Runtime Configuration](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/config.py): `src/config.py`
- [Source Reconciliation](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.31_run_source_to_database_reconciliation.py): `src/05.31_run_source_to_database_reconciliation.py`
- [Database Integrity](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/05.33_validate_database_integrity.py): `src/05.33_validate_database_integrity.py`
- [Database Backup](https://github.com/hemug1430/retail_sales_analytics/blob/main/src/10.05_backup_final_mysql_database.py): `src/10.05_backup_final_mysql_database.py`

### Other components

- **Pipeline Operator**

## Connections

- Pipeline Operator → Pipeline Runner: runs
- Pipeline Runner → File Ingestion: orchestrates
- File Ingestion → Duplicate File Detection: checks hashes
- File Ingestion → Schema Validation: validates schema
- Schema Validation → Invalid File Quarantine: routes failures
- File Ingestion → Intake Audit Logging: logs intake
- File Ingestion → Data Standardization: provides files
- Data Standardization → Cleaning Rules: applies cleaning
- Data Standardization → Staging Load: loads cleaned data
- Staging Load → Staging Validation: validates counts
- Staging Load → Dimension Merges: feeds staging
- Staging Load → Sales Incremental Load: feeds staging
- Staging Load → Returns Incremental Load: feeds staging
- Staging Load → Targets Incremental Load: feeds staging
- Dimension Merges → MySQL Warehouse: merges dimensions
- Sales Incremental Load → MySQL Warehouse: upserts sales
- Returns Incremental Load → MySQL Warehouse: loads returns
- Targets Incremental Load → MySQL Warehouse: loads targets
- SCD History Updates → MySQL Warehouse: updates history
- MySQL Warehouse → Analytical Views: exposes views
- MySQL Warehouse → Date Dimension: supports time analysis
- Analytical Views → Power BI Reports: feeds reporting
- MySQL Warehouse → Source Reconciliation: compares records (evidence: `src/05.31_run_source_to_database_reconciliation.py`)
- MySQL Warehouse → Database Integrity: checks integrity (evidence: `src/05.33_validate_database_integrity.py`)
- MySQL Warehouse → Database Backup: backs up data (evidence: `src/10.05_backup_final_mysql_database.py`)
- Runtime Configuration → File Ingestion: configures paths (evidence: `src/data_ingestion.py`)
- Runtime Configuration → Staging Validation: configures database (evidence: `src/05.17_validate_staging_row_counts.py`)
- Runtime Configuration → Returns Incremental Load: configures database (evidence: `src/05.24_run_returns_incremental_insert.py`)
