How brqno/olist-analytics-pipeline works
This is an Olist e-commerce analytics pipeline: public CSVs are loaded into PostgreSQL, transformed into analytical dimensions and facts with dbt, and exported as per-table CSVs to S3 for business analysts. Airflow orchestrates the scheduled workflow. The README documents the stage sequence and mart capabilities; dbt model internals were not sampled, so relationships within those stages are not expanded.
Main entry points: load_and_transformation.py Airflow DAG · intermediate Enrichment models · marts_to_s3.py Mart exporter · data_load.py CSV loader
Components 12
Data ingestion
- Olist CSVs
data/raw - CSV loader
load/data_load.py - Bronze tables
dbt transformation
- Staging models
olist_dbt/models/staging - Enrichment models
olist_dbt/models/intermediate - State regions seed
olist_dbt/seeds/seed_regioes_estados.csv - Dimensions
olist_dbt/models/marts - Order facts
olist_dbt/models/marts
Workflow orchestration
- Airflow DAG
airflow/dags/load_and_transformation.py
Analytics delivery
- Mart exporter
send/marts_to_s3.py - Amazon S3
Other components
- Business analyst
Connections 15
- Olist CSVs → CSV loader: reads CSVs
load/data_load.py - CSV loader → Bronze tables: loads tables
README.md - Bronze tables → Staging models: transforms source
README.md - Staging models → Enrichment models: enriches data
README.md - State regions seed → Enrichment models: enriches geolocation
README.md - Enrichment models → Dimensions: builds dimensions
README.md - Enrichment models → Order facts: builds facts
README.md - Airflow DAG → CSV loader: runs load
README.md - Airflow DAG → State regions seed: runs seed
README.md - Airflow DAG → Staging models: runs dbt
README.md - Airflow DAG → Mart exporter: runs export
README.md - Dimensions → Mart exporter: reads marts
send/marts_to_s3.py - Order facts → Mart exporter: reads marts
send/marts_to_s3.py - Mart exporter → Amazon S3: uploads CSVs
send/marts_to_s3.py - Amazon S3 → Business analyst: provides CSVs
README.md