mohammadsaif001/elt-lakehouse

Purpose and Entry Points

This repository implements a synthetic e-commerce lakehouse: generate Olist-shaped entities, stream/load raw records, transform them through Bronze/Silver/Gold layers, and publish warehouse analytics. Primary executable/orchestration entry points are generators/build_dataset.py, ingestion/bronze/run.py, ingestion/kafka/kafka-to-bronze.py, Spark jobs under spark/jobs/, and the Airflow DAG airflow/dags/ecommerce_pipeline.py.

1. Synthetic Source Dataset and Referential Pools

Generation is dependency-aware: seller/customer/product pools support orders; orders and items support payments and reviews. Profiling distributions in data/profiling/ drive realistic values, while persistent pools establish cross-entity foreign-key consistency. Outputs are JSON datasets in storage/generated/.

• generators/build_dataset.py

• generators/base/pool_manager.py

• metadata/pools/customer_pool.json

2. Event and Raw-Ingestion Boundary

Generated/source records are moved through Kafka or batch bootstrap into the Bronze/raw layer. Kafka producer/consumer/topic management are isolated from generic ingestion abstractions, allowing Bronze ingestion to be the storage boundary rather than downstream transformation logic.

• ingestion/kafka/producer.py

• ingestion/kafka/consumer.py

• ingestion/core/ingestor.py

Kafka configuration is centralized in config/kafka_config.yaml; bootstrap loaders exist for initial CSV-derived source loading under ingestion/bootstrap/.

3. Bronze Lake and Spark Processing

Bronze jobs persist raw inputs, then Spark Silver jobs clean and standardize entity datasets. Shared Spark session, schema registry, and paths form the runtime contract between jobs. Incremental support includes CDC, merge handling, and watermarks/checkpoints, so repeated processing can maintain state rather than requiring full rebuilds.

• spark/jobs/bronze_job.py

• spark/jobs/silver_job.py

• spark/incremental/merge_handler.py

State artifacts are represented in metadata/checkpoints/bronze_checkpoint.json and metadata/watermarks/watermark.json; Silver storage is rooted at storage/silver/.

4. Quality and Schema Enforcement

Quality checks are a first-class stage around Spark processing, covering schema validation, nulls, duplicates, freshness, and referential integrity. Entity contracts define expected input structures; configurable rules supply policy. Failures can be surfaced through Airflow callbacks/alerts.

• spark/jobs/quality_job.py

• spark/quality/schema_validation.py

• contracts/order_schema.json

Relevant control/configuration boundaries are config/dq_rules.yaml, airflow/include/callbacks.py, and airflow/include/alerts.py.

5. Analytics Modeling and Warehouse Publication

dbt models convert staged operational data into intermediate metrics, dimensional marts, and Gold analytics such as revenue, customer segmentation, payment analysis, and seller scorecards. PostgreSQL loader/schema scripts provide the serving warehouse boundary.

• dbt/models/staging/stg_orders.sql

• dbt/models/marts/fact_orders.sql

• dbt/models/gold/monthly_revenue.sql

Warehouse publishing is implemented through warehouse/loaders/gold_to_postgres.py, with relational schema managed in warehouse/postgres/create_tables.sql.

6. Orchestration, Runtime, and Operations

Airflow coordinates pipeline stages and invokes custom dbt/Spark operators. Docker Compose supplies the local multi-service runtime: Airflow, Spark, Kafka, and PostgreSQL. Prometheus collects metrics and Grafana visualizes Kafka, Spark, and pipeline health; GitHub workflows run CI and specialized dbt/Spark validation.

• airflow/dags/ecommerce_pipeline.py

• airflow/plugins/spark_operator.py

• docker/docker-compose.yml

Operational assets include monitoring/prometheus/prometheus.yml, monitoring/grafana/dashboards/pipeline_dashboard.json, and .github/workflows/ci.yml.