# ramijaber4/e-commerceetl architecture

> Architecture diagram of the GitHub repository ramijaber4/e-commerceetl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ramijaber4/e-commerceetl
- Repository: https://github.com/ramijaber4/e-commerceetl
- Video tour (about a minute): https://gitdiagram.com/ramijaber4/e-commerceetl/video

## Overview

This is a PySpark Customer 360 ETL application. Its main workflow reads CSV datasets, enforces schemas, cleans entity records, joins transactions with customer, country, and product data, adds business metrics, runs regional and commercial aggregations, and writes Parquet output. Reviews are extracted and cleaned but are not used in the joined output in the sampled entry point. The tree omits transactions.csv despite README references; the graph therefore treats data/ as the documented input source without asserting that file is present. Wiring reflects the sampled main.py and module calls.

Main entry points: [`src/main.py`](https://github.com/ramijaber4/e-commerceetl/blob/main/src/main.py) (ETL orchestrator), [`src/extract.py`](https://github.com/ramijaber4/e-commerceetl/blob/main/src/extract.py) (CSV extraction), [`src/transform.py`](https://github.com/ramijaber4/e-commerceetl/blob/main/src/transform.py) (Entity joins), [`src/load.py`](https://github.com/ramijaber4/e-commerceetl/blob/main/src/load.py) (Parquet loading)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Input and schemas"]
  node_input["CSV datasets"]
  node_schemas["Input schemas<br/>[schemas.py]"]
  node_extract["CSV extraction<br/>[extract.py]"]
end

subgraph group_quality["Data quality"]
  node_clean["Entity cleaning<br/>[clean.py]"]
end

subgraph group_pipeline["Customer 360 pipeline"]
  node_main["ETL orchestrator<br/>[main.py]"]
  node_join["Entity joins<br/>[transform.py]"]
  node_regional["Regional statistics<br/>[transform.py]"]
  node_rules["Business metrics<br/>[transform.py]"]
  node_analytics["Revenue and products<br/>[analytics.py]"]
  node_output["Partitioned Parquet"]
  node_load["Parquet loading<br/>[load.py]"]
end

subgraph group_runtime["Spark runtime"]
  node_config["Paths and settings<br/>[config.py]"]
  node_utils["Spark and logging<br/>[utils.py]"]
end

node_operator(("Run pipeline"))
node_spark(("Spark session"))

node_operator -->|"runs"| node_main
node_main -->|"creates session"| node_utils
node_utils -->|"returns session"| node_spark
node_main -->|"calls readers"| node_extract
node_extract -->|"reads CSVs"| node_input
node_extract -->|"applies schemas"| node_schemas
node_extract -->|"uses paths"| node_config
node_main -->|"cleans entities"| node_clean
node_main -->|"joins records"| node_join
node_main -->|"calculates stats"| node_regional
node_main -->|"adds metrics"| node_rules
node_main -->|"runs aggregations"| node_analytics
node_main -->|"loads result"| node_load
node_load -->|"writes Parquet"| node_output
node_main -->|"reads settings"| node_config
node_join -->|"consumes joined data"| node_regional
node_join -->|"consumes joined data"| node_rules
node_rules -->|"provides metrics"| node_analytics

click node_input "https://github.com/ramijaber4/e-commerceetl/tree/main/data"
click node_schemas "https://github.com/ramijaber4/e-commerceetl/blob/main/src/schemas.py"
click node_extract "https://github.com/ramijaber4/e-commerceetl/blob/main/src/extract.py"
click node_clean "https://github.com/ramijaber4/e-commerceetl/blob/main/src/clean.py"
click node_main "https://github.com/ramijaber4/e-commerceetl/blob/main/src/main.py"
click node_join "https://github.com/ramijaber4/e-commerceetl/blob/main/src/transform.py"
click node_regional "https://github.com/ramijaber4/e-commerceetl/blob/main/src/transform.py"
click node_rules "https://github.com/ramijaber4/e-commerceetl/blob/main/src/transform.py"
click node_analytics "https://github.com/ramijaber4/e-commerceetl/blob/main/src/analytics.py"
click node_load "https://github.com/ramijaber4/e-commerceetl/blob/main/src/load.py"
click node_config "https://github.com/ramijaber4/e-commerceetl/blob/main/src/config.py"
click node_utils "https://github.com/ramijaber4/e-commerceetl/blob/main/src/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_input,node_schemas,node_extract toneBlue
class node_clean toneAmber
class node_main,node_join,node_regional,node_rules,node_analytics,node_output,node_load toneMint
class node_config,node_utils toneRose
class node_operator,node_spark toneIndigo
```

## Components

### Input and schemas

- [CSV datasets](https://github.com/ramijaber4/e-commerceetl/tree/main/data): `data`
- [Input schemas](https://github.com/ramijaber4/e-commerceetl/blob/main/src/schemas.py): `src/schemas.py`
- [CSV extraction](https://github.com/ramijaber4/e-commerceetl/blob/main/src/extract.py): `src/extract.py`

### Data quality

- [Entity cleaning](https://github.com/ramijaber4/e-commerceetl/blob/main/src/clean.py): `src/clean.py`

### Customer 360 pipeline

- [ETL orchestrator](https://github.com/ramijaber4/e-commerceetl/blob/main/src/main.py): `src/main.py`
- [Entity joins](https://github.com/ramijaber4/e-commerceetl/blob/main/src/transform.py): `src/transform.py`
- [Regional statistics](https://github.com/ramijaber4/e-commerceetl/blob/main/src/transform.py): `src/transform.py`
- [Business metrics](https://github.com/ramijaber4/e-commerceetl/blob/main/src/transform.py): `src/transform.py`
- [Revenue and products](https://github.com/ramijaber4/e-commerceetl/blob/main/src/analytics.py): `src/analytics.py`
- **Partitioned Parquet**
- [Parquet loading](https://github.com/ramijaber4/e-commerceetl/blob/main/src/load.py): `src/load.py`

### Spark runtime

- [Paths and settings](https://github.com/ramijaber4/e-commerceetl/blob/main/src/config.py): `src/config.py`
- [Spark and logging](https://github.com/ramijaber4/e-commerceetl/blob/main/src/utils.py): `src/utils.py`

### Other components

- **Run pipeline**
- **Spark session**

## Connections

- Run pipeline → ETL orchestrator: runs
- ETL orchestrator → Spark and logging: creates session
- Spark and logging → Spark session: returns session
- ETL orchestrator → CSV extraction: calls readers
- CSV extraction → CSV datasets: reads CSVs
- CSV extraction → Input schemas: applies schemas
- CSV extraction → Paths and settings: uses paths
- ETL orchestrator → Entity cleaning: cleans entities
- ETL orchestrator → Entity joins: joins records
- ETL orchestrator → Regional statistics: calculates stats
- ETL orchestrator → Business metrics: adds metrics
- ETL orchestrator → Revenue and products: runs aggregations
- ETL orchestrator → Parquet loading: loads result
- Parquet loading → Partitioned Parquet: writes Parquet
- ETL orchestrator → Paths and settings: reads settings
- Entity joins → Regional statistics: consumes joined data
- Entity joins → Business metrics: consumes joined data
- Business metrics → Revenue and products: provides metrics