How ramijaber4/e-commerceetl works
This is a PySpark Customer 360 ETL application. Its main workflow reads CSV datasets, enforces schemas, cleans entity records, joins transactions with customer, country, and product data, adds business metrics, runs regional and commercial aggregations, and writes Parquet output. Reviews are extracted and cleaned but are not used in the joined output in the sampled entry point. The tree omits transactions.csv despite README references; the graph therefore treats data/ as the documented input source without asserting that file is present. Wiring reflects the sampled main.py and module calls.
Main entry points: main.py ETL orchestrator · extract.py CSV extraction · transform.py Entity joins · load.py Parquet loading
Components 15
Input and schemas
- CSV datasets
data - Input schemas
src/schemas.py - CSV extraction
src/extract.py
Data quality
- Entity cleaning
src/clean.py
Customer 360 pipeline
- ETL orchestrator
src/main.py - Entity joins
src/transform.py - Regional statistics
src/transform.py - Business metrics
src/transform.py - Revenue and products
src/analytics.py - Partitioned Parquet
- Parquet loading
src/load.py
Spark runtime
- Paths and settings
src/config.py - Spark and logging
src/utils.py
Other components
- Run pipeline
- Spark session
Connections 18
- Run pipeline → ETL orchestrator: runs
- ETL orchestrator → Spark and logging: creates session
- Spark and logging → Spark session: returns session
- ETL orchestrator → CSV extraction: calls readers
- CSV extraction → CSV datasets: reads CSVs
- CSV extraction → Input schemas: applies schemas
- CSV extraction → Paths and settings: uses paths
- ETL orchestrator → Entity cleaning: cleans entities
- ETL orchestrator → Entity joins: joins records
- ETL orchestrator → Regional statistics: calculates stats
- ETL orchestrator → Business metrics: adds metrics
- ETL orchestrator → Revenue and products: runs aggregations
- ETL orchestrator → Parquet loading: loads result
- Parquet loading → Partitioned Parquet: writes Parquet
- ETL orchestrator → Paths and settings: reads settings
- Entity joins → Regional statistics: consumes joined data
- Entity joins → Business metrics: consumes joined data
- Business metrics → Revenue and products: provides metrics