How satya9348/incremental-delta-pipeline works
This repository is a PySpark and Delta Lake case study for incrementally applying complete daily order snapshots. A snapshot is validated, deduplicated by order and timestamp (with status precedence for ties), then passed to one Delta MERGE. That merge inserts new orders, updates newer versions, ignores stale versions, reactivates returning deleted orders, and soft-deletes active orders absent from a valid snapshot. The notebook also creates the target table. The workflow is implemented in one sampled source file; the CSVs are not shown as runtime inputs, so their connection is omitted.
Main entry points: incremental_order_pipeline.py Source DataFrame
Components 7
Snapshot ingestion
- Daily snapshot
- Source DataFrame
notebooks/incremental_order_pipeline.py
Order processing
- Snapshot validation
notebooks/incremental_order_pipeline.py - Source deduplication
notebooks/incremental_order_pipeline.py - Order state MERGE
notebooks/incremental_order_pipeline.py
Delta persistence
- Target table setup
notebooks/incremental_order_pipeline.py - Delta target table
Connections 6
- Daily snapshot → Source DataFrame: provides snapshot
README.md - Source DataFrame → Snapshot validation: validates
notebooks/incremental_order_pipeline.py - Snapshot validation → Source deduplication: then calls
notebooks/incremental_order_pipeline.py - Source deduplication → Order state MERGE: passes deduplicated data
notebooks/incremental_order_pipeline.py - Target table setup → Delta target table: creates if absent
notebooks/incremental_order_pipeline.py - Order state MERGE → Delta target table: merges order state
notebooks/incremental_order_pipeline.py