How 208dongpha/ecommerce works
This is a local e-commerce lakehouse analytics pipeline. Its main workflow generates synthetic events, streams them through Bronze and Silver Delta layers, aggregates daily/category metrics into Gold, then serves those metrics through Trino and Superset. Airflow schedules event generation and requests Gold refreshes. The graph preserves README-documented integrations and capabilities; source excerpts confirm producer, streaming, transformation, and batch behavior, while the runner, Trino, and Superset links rely on README documentation.
Main entry points: ecommerce_pipeline.py Airflow DAG · streaming_job.py Bronze Stream · silver_job.py Silver Stream · batch_job.py Gold Batch Job
Components 15
Orchestration
- Airflow DAG
airflow/dags/ecommerce_pipeline.py - Gold Trigger Files
airflow/triggers
Event Ingestion
- Event Producer
producer/producer.py - Kafka
Lakehouse Processing
- Bronze Stream
spark/apps/streaming_job.py - Bronze Delta
- Silver Stream
spark/apps/silver_job.py - Silver Delta
- Gold Runner
spark/apps/gold_runner.sh - Gold Batch Job
spark/apps/batch_job.py - Gold Delta
Analytics Serving
- Trino SQL
trino/catalog/delta.properties - Superset BI
superset - Sales Dashboard
dashboards/Ecommerce Dashboardv1.html
Other components
- Pipeline Operator
Connections 17
- Pipeline Operator → Airflow DAG: triggers DAG
README.md - Airflow DAG → Event Producer: runs
airflow/dags/ecommerce_pipeline.py - Event Producer → Kafka: publishes events
producer/producer.py - Airflow DAG → Bronze Stream: allows processing
README.md - Airflow DAG → Silver Stream: waits for processing
README.md - Bronze Stream → Kafka: reads events
spark/apps/streaming_job.py - Bronze Stream → Bronze Delta: writes records
spark/apps/streaming_job.py - Silver Stream → Bronze Delta: reads records
spark/apps/silver_job.py - Silver Stream → Silver Delta: writes cleaned events
spark/apps/silver_job.py - Airflow DAG → Gold Trigger Files: writes request
airflow/dags/ecommerce_pipeline.py - Gold Trigger Files → Gold Runner: signals batch run
README.md - Gold Runner → Gold Batch Job: runs job
README.md - Gold Batch Job → Silver Delta: reads events
spark/apps/batch_job.py - Gold Batch Job → Gold Delta: writes metrics
spark/apps/batch_job.py - Trino SQL → Gold Delta: queries table
README.md - Superset BI → Trino SQL: queries metrics
README.md - Superset BI → Sales Dashboard: visualizes analytics
README.md