How rchdev1998-jpg/etl_pipeline_databricks works
This is a PySpark/Databricks lakehouse ETL project implementing a CRM/ERP medallion pipeline. Source CSVs are ingested into Bronze, Silver applies configured cleaning and SCD Type 2 or overwrite loads, and Gold builds customer/product dimensions and a sales fact for analytics. Logging is a shared operational layer. The README documents broader data-quality and analytical capabilities; the graph reflects sampled implementations and documented consumption. Relationships not visible in the sampled code are omitted.
Main entry points: bronze.py Bronze ingestion · silver.py Silver transforms · log.py ETL logging · gold.py Gold publishing
Components 14
Source ingestion
- Bronze ingestion
code/bronze.py - Ingestion settings
code/bronze_config.py - Bronze tables
code/bronze.py
Data refinement
- Silver transforms
code/silver.py - Transform settings
code/silver_config.py - Silver tables
code/silver.py
Business analytics
- Gold publishing
code/gold.py - Business models
code/gold_config.py - Gold datasets
code/gold.py
Pipeline operations
- ETL logging
code/log.py - ETL log tables
code/log.py
Other components
- CRM files
- ERP files
- Analysts / BI
Connections 15
- CRM files → Bronze ingestion: provides CSVs
code/bronze.py - ERP files → Bronze ingestion: provides CSVs
code/bronze.py - Bronze ingestion → Ingestion settings: loads settings
code/bronze.py - Bronze ingestion → Bronze tables: writes tables
code/bronze.py - Bronze ingestion → ETL logging: records runs
code/bronze.py - Silver transforms → Transform settings: loads settings
code/silver.py - Silver transforms → Bronze tables: queries sources
- Silver transforms → Silver tables: loads tables
code/silver.py - Silver transforms → ETL logging: records runs
code/silver.py - Gold publishing → Business models: loads settings
code/gold.py - Business models → Silver tables: queries and joins
code/gold_config.py - Gold publishing → Gold datasets: publishes tables
code/gold.py - Gold publishing → ETL logging: records runs
code/gold.py - ETL logging → ETL log tables: inserts summaries
code/log.py - Gold datasets → Analysts / BI: serves analytics
README.md