# antonyremay/olist-etl-project architecture

> Architecture diagram of the GitHub repository antonyremay/olist-etl-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/antonyremay/olist-etl-project
- Repository: https://github.com/antonyremay/olist-etl-project

## Overview

Olist is an e-commerce data engineering and sales-insights project. Its main workflow takes source CSVs through per-dataset Pandas cleaning and validation, writes cleaned CSVs, then supports MySQL loading and Spark analysis; results also feed SQL business analysis and a Power BI dashboard. Separate local PySpark and Databricks branches demonstrate distributed processing, including seller aggregates and an orders/items join. The graph preserves the documented stages, but the SQL queries and dashboard internals are not represented by sampled source, so those relationships rely on the README.

Main entry points: [`Data_sets/Cleaned_Dataset`](https://github.com/antonyremay/olist-etl-project/tree/main/Data_sets/Cleaned_Dataset) (Cleaned CSVs), [`python_scripts/customer_data.py`](https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/customer_data.py) (Customer ETL), [`python_scripts/geo_map.py`](https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/geo_map.py) (Geolocation ETL), [`python_scripts/items_data.py`](https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/items_data.py) (Order Items ETL)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_etl["Python ETL"]
  node_customers["Customer ETL<br/>[customer_data.py]"]
  node_geolocation["Geolocation ETL<br/>[geo_map.py]"]
  node_items["Order Items ETL<br/>[items_data.py]"]
  node_orders["Orders ETL<br/>[olist_order.py]"]
  node_payments["Payments ETL<br/>[order_payments.py]"]
  node_categories["Category ETL"]
  node_products["Products ETL<br/>[products_data.py]"]
  node_reviews["Reviews ETL<br/>[reviews_data.py]"]
  node_sellers["Sellers ETL<br/>[sellers_data.py]"]
end

subgraph group_storage["Data and Database"]
  node_cleaned[("Cleaned CSVs")]
  node_mysql_loader["MySQL Loader"]
  node_mysql[("MySQL")]
end

subgraph group_spark["Spark Analytics"]
  node_spark_read["Read Items"]
  node_spark_quality["Items Quality"]
  node_spark_sales["Seller Sales"]
  node_spark_join["Orders Join"]
  node_databricks["Databricks Analysis"]
end

subgraph group_insights["Business Insights"]
  node_sql["SQL Analysis"]
  node_dashboard["Power BI Dashboard"]
end

node_source["Olist CSVs"]

node_source -->|"loads and cleans"| node_customers
node_source -->|"loads and cleans"| node_geolocation
node_source -->|"loads and cleans"| node_items
node_source -->|"loads and cleans"| node_orders
node_source -->|"loads and cleans"| node_payments
node_source -->|"loads and cleans"| node_categories
node_source -->|"loads and cleans"| node_products
node_source -->|"loads and cleans"| node_reviews
node_source -->|"loads and cleans"| node_sellers
node_customers -->|"writes"| node_cleaned
node_geolocation -->|"writes"| node_cleaned
node_items -->|"writes"| node_cleaned
node_orders -->|"writes"| node_cleaned
node_payments -->|"writes"| node_cleaned
node_categories -->|"writes"| node_cleaned
node_products -->|"writes"| node_cleaned
node_reviews -->|"writes"| node_cleaned
node_sellers -->|"writes"| node_cleaned
node_cleaned -->|"loads from"| node_mysql_loader
node_mysql_loader -->|"loads tables"| node_mysql
node_cleaned -->|"reads"| node_spark_read
node_cleaned -->|"reads"| node_spark_quality
node_cleaned -->|"reads"| node_spark_sales
node_cleaned -->|"reads"| node_spark_join
node_databricks -->|"saves tables for"| node_sql
node_mysql -->|"supports queries"| node_sql
node_sql -.->|"informs"| node_dashboard
node_mysql -.->|"visualizes sales insights"| node_dashboard

click node_customers "https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/customer_data.py"
click node_geolocation "https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/geo_map.py"
click node_items "https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/items_data.py"
click node_orders "https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/olist_order.py"
click node_payments "https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/order_payments.py"
click node_categories "https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/prod_category_translate_data.py"
click node_products "https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/products_data.py"
click node_reviews "https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/reviews_data.py"
click node_sellers "https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/sellers_data.py"
click node_cleaned "https://github.com/antonyremay/olist-etl-project/tree/main/Data_sets/Cleaned_Dataset"
click node_mysql_loader "https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/load_data_to_mysql.py"
click node_spark_read "https://github.com/antonyremay/olist-etl-project/blob/main/pyspark_scripts/01_read_order_items.py"
click node_spark_quality "https://github.com/antonyremay/olist-etl-project/blob/main/pyspark_scripts/02_order_items_quality.py"
click node_spark_sales "https://github.com/antonyremay/olist-etl-project/blob/main/pyspark_scripts/03_seller_sales_analysis.py"
click node_spark_join "https://github.com/antonyremay/olist-etl-project/blob/main/pyspark_scripts/04_orders_items_join.py"
click node_databricks "https://github.com/antonyremay/olist-etl-project/blob/main/databricks_notebooks/Olist_PySpark_Analysis.py"
click node_dashboard "https://github.com/antonyremay/olist-etl-project/blob/main/olist_powerbi_dashboard.pbix"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_customers,node_geolocation,node_items,node_orders,node_payments,node_categories,node_products,node_reviews,node_sellers toneBlue
class node_cleaned,node_mysql_loader,node_mysql toneAmber
class node_spark_read,node_spark_quality,node_spark_sales,node_spark_join,node_databricks toneMint
class node_sql,node_dashboard toneRose
class node_source toneIndigo
```

## Components

### Python ETL

- [Customer ETL](https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/customer_data.py): `python_scripts/customer_data.py`
- [Geolocation ETL](https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/geo_map.py): `python_scripts/geo_map.py`
- [Order Items ETL](https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/items_data.py): `python_scripts/items_data.py`
- [Orders ETL](https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/olist_order.py): `python_scripts/olist_order.py`
- [Payments ETL](https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/order_payments.py): `python_scripts/order_payments.py`
- [Category ETL](https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/prod_category_translate_data.py): `python_scripts/prod_category_translate_data.py`
- [Products ETL](https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/products_data.py): `python_scripts/products_data.py`
- [Reviews ETL](https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/reviews_data.py): `python_scripts/reviews_data.py`
- [Sellers ETL](https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/sellers_data.py): `python_scripts/sellers_data.py`

### Data and Database

- [Cleaned CSVs](https://github.com/antonyremay/olist-etl-project/tree/main/Data_sets/Cleaned_Dataset): `Data_sets/Cleaned_Dataset`
- [MySQL Loader](https://github.com/antonyremay/olist-etl-project/blob/main/python_scripts/load_data_to_mysql.py): `python_scripts/load_data_to_mysql.py`
- **MySQL**

### Spark Analytics

- [Read Items](https://github.com/antonyremay/olist-etl-project/blob/main/pyspark_scripts/01_read_order_items.py): `pyspark_scripts/01_read_order_items.py`
- [Items Quality](https://github.com/antonyremay/olist-etl-project/blob/main/pyspark_scripts/02_order_items_quality.py): `pyspark_scripts/02_order_items_quality.py`
- [Seller Sales](https://github.com/antonyremay/olist-etl-project/blob/main/pyspark_scripts/03_seller_sales_analysis.py): `pyspark_scripts/03_seller_sales_analysis.py`
- [Orders Join](https://github.com/antonyremay/olist-etl-project/blob/main/pyspark_scripts/04_orders_items_join.py): `pyspark_scripts/04_orders_items_join.py`
- [Databricks Analysis](https://github.com/antonyremay/olist-etl-project/blob/main/databricks_notebooks/Olist_PySpark_Analysis.py): `databricks_notebooks/Olist_PySpark_Analysis.py`

### Business Insights

- **SQL Analysis**
- [Power BI Dashboard](https://github.com/antonyremay/olist-etl-project/blob/main/olist_powerbi_dashboard.pbix): `olist_powerbi_dashboard.pbix`

### Other components

- **Olist CSVs**

## Connections

- Olist CSVs → Customer ETL: loads and cleans (evidence: `README.md`)
- Olist CSVs → Geolocation ETL: loads and cleans (evidence: `README.md`)
- Olist CSVs → Order Items ETL: loads and cleans (evidence: `README.md`)
- Olist CSVs → Orders ETL: loads and cleans (evidence: `README.md`)
- Olist CSVs → Payments ETL: loads and cleans (evidence: `README.md`)
- Olist CSVs → Category ETL: loads and cleans (evidence: `README.md`)
- Olist CSVs → Products ETL: loads and cleans (evidence: `README.md`)
- Olist CSVs → Reviews ETL: loads and cleans (evidence: `README.md`)
- Olist CSVs → Sellers ETL: loads and cleans (evidence: `README.md`)
- Customer ETL → Cleaned CSVs: writes (evidence: `README.md`)
- Geolocation ETL → Cleaned CSVs: writes (evidence: `python_scripts/geo_map.py`)
- Order Items ETL → Cleaned CSVs: writes (evidence: `README.md`)
- Orders ETL → Cleaned CSVs: writes (evidence: `python_scripts/olist_order.py`)
- Payments ETL → Cleaned CSVs: writes (evidence: `README.md`)
- Category ETL → Cleaned CSVs: writes (evidence: `python_scripts/prod_category_translate_data.py`)
- Products ETL → Cleaned CSVs: writes (evidence: `python_scripts/products_data.py`)
- Reviews ETL → Cleaned CSVs: writes (evidence: `python_scripts/reviews_data.py`)
- Sellers ETL → Cleaned CSVs: writes (evidence: `python_scripts/sellers_data.py`)
- Cleaned CSVs → MySQL Loader: loads from (evidence: `README.md`)
- MySQL Loader → MySQL: loads tables (evidence: `README.md`)
- Cleaned CSVs → Read Items: reads (evidence: `pyspark_scripts/01_read_order_items.py`)
- Cleaned CSVs → Items Quality: reads (evidence: `pyspark_scripts/02_order_items_quality.py`)
- Cleaned CSVs → Seller Sales: reads (evidence: `pyspark_scripts/03_seller_sales_analysis.py`)
- Cleaned CSVs → Orders Join: reads (evidence: `pyspark_scripts/04_orders_items_join.py`)
- Databricks Analysis → SQL Analysis: saves tables for (evidence: `databricks_notebooks/Olist_PySpark_Analysis.py`)
- MySQL → SQL Analysis: supports queries (evidence: `README.md`)
- SQL Analysis → Power BI Dashboard: informs
- MySQL → Power BI Dashboard: visualizes sales insights (evidence: `README.md`)