# antonyremay/retail-etl-pipeline architecture

> Architecture diagram of the GitHub repository antonyremay/retail-etl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/antonyremay/retail-etl-pipeline
- Repository: https://github.com/antonyremay/retail-etl-pipeline

## Overview

This repository is a retail-sales ETL pipeline paired with a Power BI analytics dashboard. Its main workflow reads transaction data, cleans and validates it, loads rows into MySQL, then reconciles source and target counts and amounts. The documented analytics branch uses Power BI to explore sales trends, products, categories, and cities. The ETL stages are visible in one script; the Power BI query and dashboard implementation were not sampled, so their connections are shown only where the README documents them.

Main entry points: [`etl_pipeline.py`](https://github.com/antonyremay/retail-etl-pipeline/blob/main/etl_pipeline.py) (Clean and transform), [`powerbi/mysql_fact_retail_sales.pq`](https://github.com/antonyremay/retail-etl-pipeline/blob/main/powerbi/mysql_fact_retail_sales.pq) (Sales data query), [`dashboard.md`](https://github.com/antonyremay/retail-etl-pipeline/blob/main/dashboard.md) (Sales dashboard)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_etl["ETL Pipeline"]
  node_extract["Extract transactions<br/>[etl_pipeline.py]"]
  node_transform["Clean and transform<br/>[etl_pipeline.py]"]
  node_validate["Validate records<br/>[etl_pipeline.py]"]
  node_load["Load fact rows<br/>[etl_pipeline.py]"]
  node_reconcile["Reconcile totals<br/>[etl_pipeline.py]"]
end

subgraph group_analytics["Sales Analytics"]
  node_powerbi_query["Sales data query"]
  node_dashboard["Sales dashboard<br/>[dashboard.md]"]
end

node_operator(("Pipeline operator"))
node_source["Transaction source"]
node_mysql[("MySQL fact table")]
node_powerbi_service["Power BI report"]

node_operator -->|"runs"| node_extract
node_source -->|"reads"| node_extract
node_extract -->|"cleans"| node_transform
node_transform -->|"validates"| node_validate
node_transform -->|"loads"| node_load
node_load -->|"inserts rows"| node_mysql
node_transform -->|"compares source"| node_reconcile
node_reconcile -->|"queries totals"| node_mysql
node_mysql -.->|"provides sales data"| node_powerbi_query
node_powerbi_query -.->|"supports analysis"| node_dashboard
node_dashboard -->|"published as"| node_powerbi_service

click node_extract "https://github.com/antonyremay/retail-etl-pipeline/blob/main/etl_pipeline.py"
click node_transform "https://github.com/antonyremay/retail-etl-pipeline/blob/main/etl_pipeline.py"
click node_validate "https://github.com/antonyremay/retail-etl-pipeline/blob/main/etl_pipeline.py"
click node_load "https://github.com/antonyremay/retail-etl-pipeline/blob/main/etl_pipeline.py"
click node_reconcile "https://github.com/antonyremay/retail-etl-pipeline/blob/main/etl_pipeline.py"
click node_powerbi_query "https://github.com/antonyremay/retail-etl-pipeline/blob/main/powerbi/mysql_fact_retail_sales.pq"
click node_dashboard "https://github.com/antonyremay/retail-etl-pipeline/blob/main/dashboard.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_extract,node_transform,node_validate,node_load,node_reconcile toneBlue
class node_powerbi_query,node_dashboard,node_mysql toneAmber
class node_operator,node_source,node_powerbi_service toneIndigo
```

## Components

### ETL Pipeline

- [Extract transactions](https://github.com/antonyremay/retail-etl-pipeline/blob/main/etl_pipeline.py): `etl_pipeline.py`
- [Clean and transform](https://github.com/antonyremay/retail-etl-pipeline/blob/main/etl_pipeline.py): `etl_pipeline.py`
- [Validate records](https://github.com/antonyremay/retail-etl-pipeline/blob/main/etl_pipeline.py): `etl_pipeline.py`
- [Load fact rows](https://github.com/antonyremay/retail-etl-pipeline/blob/main/etl_pipeline.py): `etl_pipeline.py`
- [Reconcile totals](https://github.com/antonyremay/retail-etl-pipeline/blob/main/etl_pipeline.py): `etl_pipeline.py`

### Sales Analytics

- [Sales data query](https://github.com/antonyremay/retail-etl-pipeline/blob/main/powerbi/mysql_fact_retail_sales.pq): `powerbi/mysql_fact_retail_sales.pq`
- [Sales dashboard](https://github.com/antonyremay/retail-etl-pipeline/blob/main/dashboard.md): `dashboard.md`

### Other components

- **Pipeline operator**
- **Transaction source**
- **MySQL fact table**
- **Power BI report**

## Connections

- Pipeline operator → Extract transactions: runs (evidence: `README.md`)
- Transaction source → Extract transactions: reads (evidence: `etl_pipeline.py`)
- Extract transactions → Clean and transform: cleans (evidence: `etl_pipeline.py`)
- Clean and transform → Validate records: validates (evidence: `etl_pipeline.py`)
- Clean and transform → Load fact rows: loads (evidence: `etl_pipeline.py`)
- Load fact rows → MySQL fact table: inserts rows (evidence: `etl_pipeline.py`)
- Clean and transform → Reconcile totals: compares source (evidence: `etl_pipeline.py`)
- Reconcile totals → MySQL fact table: queries totals (evidence: `etl_pipeline.py`)
- MySQL fact table → Sales data query: provides sales data (evidence: `README.md`)
- Sales data query → Sales dashboard: supports analysis (evidence: `README.md`)
- Sales dashboard → Power BI report: published as (evidence: `README.md`)