How amidarkcoder/medallion-orders-pipeline works
This repository implements a scheduled customer-order analytics pipeline: Airflow runs synthetic order generation, incremental bronze ingestion, silver cleanup, and gold aggregation against PostgreSQL. Gold produces sales, logistics, and buyer marts. The Superset configuration indicates a separate dashboarding capability, but the empty README and unsampled deployment files leave its connection to the pipeline unverified.
Main entry points: customer_data_pipeline.py Airflow DAG · moke_data.py Order Generator · raw.py Bronze Ingestion · silver.py Silver Cleanup
Components 8
Orchestration
- Airflow DAG
customer_data_pipeline.py
Order Data
- Order Generator
moke_data.py - PostgreSQL
Analytics Warehouse
- Bronze Ingestion
raw.py - Silver Cleanup
silver.py - Gold Marts
gold.py
Visualization
- Superset Config
superset_config.py - Superset
Connections 10
- Airflow DAG → Order Generator: runs first
customer_data_pipeline.py - Airflow DAG → Bronze Ingestion: runs second
customer_data_pipeline.py - Airflow DAG → Silver Cleanup: runs third
customer_data_pipeline.py - Airflow DAG → Gold Marts: runs fourth
customer_data_pipeline.py - Order Generator → PostgreSQL: writes orders
moke_data.py - Airflow DAG → PostgreSQL: gets connection
customer_data_pipeline.py - Bronze Ingestion → PostgreSQL: writes bronze
raw.py - Silver Cleanup → PostgreSQL: reads bronze; writes silver
silver.py - Gold Marts → PostgreSQL: reads silver; writes marts
gold.py - Superset Config → Superset: configures