# satya9348/incremental-delta-pipeline architecture

> Architecture diagram of the GitHub repository satya9348/incremental-delta-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/satya9348/incremental-delta-pipeline
- Repository: https://github.com/satya9348/incremental-delta-pipeline

## Overview

This repository is a PySpark and Delta Lake case study for incrementally applying complete daily order snapshots. A snapshot is validated, deduplicated by order and timestamp (with status precedence for ties), then passed to one Delta MERGE. That merge inserts new orders, updates newer versions, ignores stale versions, reactivates returning deleted orders, and soft-deletes active orders absent from a valid snapshot. The notebook also creates the target table. The workflow is implemented in one sampled source file; the CSVs are not shown as runtime inputs, so their connection is omitted.

Main entry points: [`notebooks/incremental_order_pipeline.py`](https://github.com/satya9348/incremental-delta-pipeline/blob/main/notebooks/incremental_order_pipeline.py) (Source DataFrame)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Snapshot ingestion"]
  node_snapshot(("Daily snapshot"))
  node_source_df["Source DataFrame"]
end

subgraph group_processing["Order processing"]
  node_validate["Snapshot validation"]
  node_deduplicate["Source deduplication"]
  node_merge["Order state MERGE"]
end

subgraph group_persistence["Delta persistence"]
  node_create_table["Target table setup"]
  node_delta_table[("Delta target table")]
end

node_snapshot -->|"provides snapshot"| node_source_df
node_source_df -->|"validates"| node_validate
node_validate -->|"then calls"| node_deduplicate
node_deduplicate -->|"passes deduplicated data"| node_merge
node_create_table -->|"creates if absent"| node_delta_table
node_merge -->|"merges order state"| node_delta_table

click node_source_df "https://github.com/satya9348/incremental-delta-pipeline/blob/main/notebooks/incremental_order_pipeline.py"
click node_validate "https://github.com/satya9348/incremental-delta-pipeline/blob/main/notebooks/incremental_order_pipeline.py"
click node_deduplicate "https://github.com/satya9348/incremental-delta-pipeline/blob/main/notebooks/incremental_order_pipeline.py"
click node_merge "https://github.com/satya9348/incremental-delta-pipeline/blob/main/notebooks/incremental_order_pipeline.py"
click node_create_table "https://github.com/satya9348/incremental-delta-pipeline/blob/main/notebooks/incremental_order_pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_snapshot,node_source_df toneBlue
class node_validate,node_deduplicate,node_merge toneAmber
class node_create_table,node_delta_table toneMint
```

## Components

### Snapshot ingestion

- **Daily snapshot**
- [Source DataFrame](https://github.com/satya9348/incremental-delta-pipeline/blob/main/notebooks/incremental_order_pipeline.py): `notebooks/incremental_order_pipeline.py`

### Order processing

- [Snapshot validation](https://github.com/satya9348/incremental-delta-pipeline/blob/main/notebooks/incremental_order_pipeline.py): `notebooks/incremental_order_pipeline.py`
- [Source deduplication](https://github.com/satya9348/incremental-delta-pipeline/blob/main/notebooks/incremental_order_pipeline.py): `notebooks/incremental_order_pipeline.py`
- [Order state MERGE](https://github.com/satya9348/incremental-delta-pipeline/blob/main/notebooks/incremental_order_pipeline.py): `notebooks/incremental_order_pipeline.py`

### Delta persistence

- [Target table setup](https://github.com/satya9348/incremental-delta-pipeline/blob/main/notebooks/incremental_order_pipeline.py): `notebooks/incremental_order_pipeline.py`
- **Delta target table**

## Connections

- Daily snapshot → Source DataFrame: provides snapshot (evidence: `README.md`)
- Source DataFrame → Snapshot validation: validates (evidence: `notebooks/incremental_order_pipeline.py`)
- Snapshot validation → Source deduplication: then calls (evidence: `notebooks/incremental_order_pipeline.py`)
- Source deduplication → Order state MERGE: passes deduplicated data (evidence: `notebooks/incremental_order_pipeline.py`)
- Target table setup → Delta target table: creates if absent (evidence: `notebooks/incremental_order_pipeline.py`)
- Order state MERGE → Delta target table: merges order state (evidence: `notebooks/incremental_order_pipeline.py`)
