How antonyremay/olist-etl-project works
Olist is an e-commerce data engineering and sales-insights project. Its main workflow takes source CSVs through per-dataset Pandas cleaning and validation, writes cleaned CSVs, then supports MySQL loading and Spark analysis; results also feed SQL business analysis and a Power BI dashboard. Separate local PySpark and Databricks branches demonstrate distributed processing, including seller aggregates and an orders/items join. The graph preserves the documented stages, but the SQL queries and dashboard internals are not represented by sampled source, so those relationships rely on the README.
Main entry points: Cleaned_Dataset Cleaned CSVs · customer_data.py Customer ETL · geo_map.py Geolocation ETL · items_data.py Order Items ETL
Components 20
Python ETL
- Customer ETL
python_scripts/customer_data.py - Geolocation ETL
python_scripts/geo_map.py - Order Items ETL
python_scripts/items_data.py - Orders ETL
python_scripts/olist_order.py - Payments ETL
python_scripts/order_payments.py - Category ETL
python_scripts/prod_category_translate_data.py - Products ETL
python_scripts/products_data.py - Reviews ETL
python_scripts/reviews_data.py - Sellers ETL
python_scripts/sellers_data.py
Data and Database
- Cleaned CSVs
Data_sets/Cleaned_Dataset - MySQL Loader
python_scripts/load_data_to_mysql.py - MySQL
Spark Analytics
- Read Items
pyspark_scripts/01_read_order_items.py - Items Quality
pyspark_scripts/02_order_items_quality.py - Seller Sales
pyspark_scripts/03_seller_sales_analysis.py - Orders Join
pyspark_scripts/04_orders_items_join.py - Databricks Analysis
databricks_notebooks/Olist_PySpark_Analysis.py
Business Insights
- SQL Analysis
- Power BI Dashboard
olist_powerbi_dashboard.pbix
Other components
- Olist CSVs
Connections 28
- Olist CSVs → Customer ETL: loads and cleans
README.md - Olist CSVs → Geolocation ETL: loads and cleans
README.md - Olist CSVs → Order Items ETL: loads and cleans
README.md - Olist CSVs → Orders ETL: loads and cleans
README.md - Olist CSVs → Payments ETL: loads and cleans
README.md - Olist CSVs → Category ETL: loads and cleans
README.md - Olist CSVs → Products ETL: loads and cleans
README.md - Olist CSVs → Reviews ETL: loads and cleans
README.md - Olist CSVs → Sellers ETL: loads and cleans
README.md - Customer ETL → Cleaned CSVs: writes
README.md - Geolocation ETL → Cleaned CSVs: writes
python_scripts/geo_map.py - Order Items ETL → Cleaned CSVs: writes
README.md - Orders ETL → Cleaned CSVs: writes
python_scripts/olist_order.py - Payments ETL → Cleaned CSVs: writes
README.md - Category ETL → Cleaned CSVs: writes
python_scripts/prod_category_translate_data.py - Products ETL → Cleaned CSVs: writes
python_scripts/products_data.py - Reviews ETL → Cleaned CSVs: writes
python_scripts/reviews_data.py - Sellers ETL → Cleaned CSVs: writes
python_scripts/sellers_data.py - Cleaned CSVs → MySQL Loader: loads from
README.md - MySQL Loader → MySQL: loads tables
README.md - Cleaned CSVs → Read Items: reads
pyspark_scripts/01_read_order_items.py - Cleaned CSVs → Items Quality: reads
pyspark_scripts/02_order_items_quality.py - Cleaned CSVs → Seller Sales: reads
pyspark_scripts/03_seller_sales_analysis.py - Cleaned CSVs → Orders Join: reads
pyspark_scripts/04_orders_items_join.py - Databricks Analysis → SQL Analysis: saves tables for
databricks_notebooks/Olist_PySpark_Analysis.py - MySQL → SQL Analysis: supports queries
README.md - SQL Analysis → Power BI Dashboard: informs
- MySQL → Power BI Dashboard: visualizes sales insights
README.md