# arunnegi112/olist-e2e-analytics architecture

> Architecture diagram of the GitHub repository arunnegi112/olist-e2e-analytics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arunnegi112/olist-e2e-analytics
- Repository: https://github.com/arunnegi112/olist-e2e-analytics

## Overview

This repository is an end-to-end Olist e-commerce analytics project. Its main workflow cleans raw CSVs in a Python notebook, loads relational staging tables in PostgreSQL, transforms them into an analytics model, and serves that model in a three-page Power BI report. A separate EDA notebook explores staging data; SQL views provide date-limit helpers in both schemas. The README documents these relationships, but the sampled source excerpt contains only requirements, so implementation details beyond that documentation are not verified.

Main entry points: [`sql/02_Create_StagingTables.sql`](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/02_Create_StagingTables.sql) (Staging tables), [`sql/04_Create_AnalyticsTables.sql`](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/04_Create_AnalyticsTables.sql) (Dimensional model), [`sql/03_InsertData.sql`](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/03_InsertData.sql) (CSV staging load), [`notebooks/01_Data_Cleaning.ipynb`](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/notebooks/01_Data_Cleaning.ipynb) (Data cleaning)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_cleaning["Data cleaning"]
  node_cleaning_notebook["Data cleaning"]
  node_cleaned_csvs["Cleaned CSVs"]
end

subgraph group_warehouse["PostgreSQL warehouse"]
  node_database[("PostgreSQL<br/>[01_Create_db.sql]")]
  node_schemas["Warehouse schemas"]
  node_staging_tables[("Staging tables")]
  node_staging_load["CSV staging load<br/>[03_InsertData.sql]"]
  node_staging_date_view["Staging date view<br/>[05_CreateViews.sql]"]
  node_analytics_tables[("Dimensional model")]
  node_analytics_load["Analytics transform<br/>[06_InsertData.sql]"]
  node_analytics_date_view["Analytics date view"]
end

subgraph group_exploration["Staging exploration"]
  node_staging_eda["Staging EDA<br/>[02_BasicEDA.ipynb]"]
end

subgraph group_reporting["Business intelligence"]
  node_powerbi_report["Power BI dashboard"]
  node_report_measures["Report measures<br/>[dax.md]"]
end

node_raw_csvs["Raw Olist CSVs"]

node_raw_csvs -->|"profiles and cleans"| node_cleaning_notebook
node_cleaning_notebook -->|"writes cleaned data"| node_cleaned_csvs
node_cleaned_csvs -->|"loads cleaned CSVs"| node_staging_load
node_raw_csvs -->|"loads source CSVs"| node_staging_load
node_database -->|"hosts"| node_schemas
node_schemas -->|"defines staging"| node_staging_tables
node_staging_load -->|"loads rows"| node_staging_tables
node_staging_tables -->|"explores"| node_staging_eda
node_staging_tables -->|"defines date limits"| node_staging_date_view
node_staging_tables -->|"transforms from staging"| node_analytics_load
node_analytics_load -->|"populates model"| node_analytics_tables
node_analytics_tables -->|"defines date limits"| node_analytics_date_view
node_analytics_tables -->|"feeds dashboard"| node_powerbi_report
node_report_measures -->|"documents measures"| node_powerbi_report

click node_cleaning_notebook "https://github.com/arunnegi112/olist-e2e-analytics/blob/main/notebooks/01_Data_Cleaning.ipynb"
click node_database "https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/01_Create_db.sql"
click node_schemas "https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/02_Create_Schemas.sql"
click node_staging_tables "https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/02_Create_StagingTables.sql"
click node_staging_load "https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/03_InsertData.sql"
click node_staging_eda "https://github.com/arunnegi112/olist-e2e-analytics/blob/main/notebooks/02_BasicEDA.ipynb"
click node_staging_date_view "https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/05_CreateViews.sql"
click node_analytics_tables "https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/04_Create_AnalyticsTables.sql"
click node_analytics_load "https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/06_InsertData.sql"
click node_analytics_date_view "https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/07_AnalyticViews.sql"
click node_powerbi_report "https://github.com/arunnegi112/olist-e2e-analytics/blob/main/power_bi/AnalysisReport.pbix"
click node_report_measures "https://github.com/arunnegi112/olist-e2e-analytics/blob/main/docs/dax.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cleaning_notebook,node_cleaned_csvs toneBlue
class node_database,node_schemas,node_staging_tables,node_staging_load,node_staging_date_view,node_analytics_tables,node_analytics_load,node_analytics_date_view toneAmber
class node_staging_eda toneMint
class node_powerbi_report,node_report_measures toneRose
class node_raw_csvs toneIndigo
```

## Components

### Data cleaning

- [Data cleaning](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/notebooks/01_Data_Cleaning.ipynb): `notebooks/01_Data_Cleaning.ipynb`
- **Cleaned CSVs**

### PostgreSQL warehouse

- [PostgreSQL](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/01_Create_db.sql): `sql/01_Create_db.sql`
- [Warehouse schemas](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/02_Create_Schemas.sql): `sql/02_Create_Schemas.sql`
- [Staging tables](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/02_Create_StagingTables.sql): `sql/02_Create_StagingTables.sql`
- [CSV staging load](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/03_InsertData.sql): `sql/03_InsertData.sql`
- [Staging date view](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/05_CreateViews.sql): `sql/05_CreateViews.sql`
- [Dimensional model](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/04_Create_AnalyticsTables.sql): `sql/04_Create_AnalyticsTables.sql`
- [Analytics transform](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/06_InsertData.sql): `sql/06_InsertData.sql`
- [Analytics date view](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/sql/07_AnalyticViews.sql): `sql/07_AnalyticViews.sql`

### Staging exploration

- [Staging EDA](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/notebooks/02_BasicEDA.ipynb): `notebooks/02_BasicEDA.ipynb`

### Business intelligence

- [Power BI dashboard](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/power_bi/AnalysisReport.pbix): `power_bi/AnalysisReport.pbix`
- [Report measures](https://github.com/arunnegi112/olist-e2e-analytics/blob/main/docs/dax.md): `docs/dax.md`

### Other components

- **Raw Olist CSVs**

## Connections

- Raw Olist CSVs → Data cleaning: profiles and cleans (evidence: `README.md`)
- Data cleaning → Cleaned CSVs: writes cleaned data (evidence: `README.md`)
- Cleaned CSVs → CSV staging load: loads cleaned CSVs (evidence: `README.md`)
- Raw Olist CSVs → CSV staging load: loads source CSVs (evidence: `README.md`)
- PostgreSQL → Warehouse schemas: hosts (evidence: `README.md`)
- Warehouse schemas → Staging tables: defines staging (evidence: `README.md`)
- CSV staging load → Staging tables: loads rows (evidence: `README.md`)
- Staging tables → Staging EDA: explores (evidence: `README.md`)
- Staging tables → Staging date view: defines date limits (evidence: `README.md`)
- Staging tables → Analytics transform: transforms from staging (evidence: `README.md`)
- Analytics transform → Dimensional model: populates model (evidence: `README.md`)
- Dimensional model → Analytics date view: defines date limits (evidence: `README.md`)
- Dimensional model → Power BI dashboard: feeds dashboard (evidence: `README.md`)
- Report measures → Power BI dashboard: documents measures (evidence: `README.md`)
