# sxijyoti/incremental-lakehouse-cdc architecture

> Architecture diagram of the GitHub repository sxijyoti/incremental-lakehouse-cdc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sxijyoti/incremental-lakehouse-cdc
- Repository: https://github.com/sxijyoti/incremental-lakehouse-cdc

## Overview

Incremental Lakehouse CDC streams change events from Kafka through Spark Structured Streaming into transactional Iceberg tables. The consumer parses events and delegates micro-batch application to a merge handler, which deduplicates by ID and applies inserts, updates, and deletes. The project also exposes snapshot time travel, incremental reads, table maintenance, and benchmark utilities. Spark is the shared runtime/configuration layer for the catalog and object storage. Some documented capabilities, including benchmarks, have no sampled implementation excerpt, so their connections are not asserted.

Main entry points: [`src/ingestion/streaming_consumer.py`](https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/ingestion/streaming_consumer.py) (Streaming consumer), [`src/lakehouse/iceberg_operations.py`](https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/lakehouse/iceberg_operations.py) (Table operations), [`src/lakehouse/spark_session.py`](https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/lakehouse/spark_session.py) (Spark session), [`src/lakehouse/time_travel.py`](https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/lakehouse/time_travel.py) (Snapshot time travel)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["CDC ingestion"]
  node_producer["Kafka producer<br/>[kafka_producer.py]"]
  node_kafka["Kafka topics"]
  node_consumer["Streaming consumer"]
end

subgraph group_processing["Spark processing"]
  node_spark["Spark session<br/>[spark_session.py]"]
  node_merge["CDC merge handler<br/>[merge_handler.py]"]
end

subgraph group_lakehouse["Iceberg lakehouse"]
  node_icebergops["Table operations"]
  node_tables[("Iceberg tables")]
  node_catalog["Iceberg REST catalog"]
  node_storage[("MinIO storage")]
  node_time["Snapshot time travel<br/>[time_travel.py]"]
end

subgraph group_operations["Lakehouse operations"]
  node_maintenance["Table maintenance<br/>[compaction.py]"]
  node_benchmarks["Performance benchmarks"]
end

node_sources(("CDC sources"))
node_consumers(("Query consumers"))

node_sources -->|"generates events"| node_producer
node_producer -->|"publishes events"| node_kafka
node_kafka -->|"streams events"| node_consumer
node_consumer -->|"delegates batches"| node_merge
node_consumer -->|"creates tables"| node_icebergops
node_merge -->|"merges changes"| node_tables
node_spark -->|"configures connection"| node_catalog
node_spark -->|"configures access"| node_storage
node_icebergops -->|"creates and reads"| node_tables
node_time -->|"queries snapshots"| node_tables
node_time -->|"creates demo table"| node_icebergops
node_maintenance -->|"optimizes and cleans"| node_tables
node_tables -->|"serves analytics"| node_consumers

click node_producer "https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/ingestion/kafka_producer.py"
click node_consumer "https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/ingestion/streaming_consumer.py"
click node_spark "https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/lakehouse/spark_session.py"
click node_merge "https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/lakehouse/merge_handler.py"
click node_icebergops "https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/lakehouse/iceberg_operations.py"
click node_time "https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/lakehouse/time_travel.py"
click node_maintenance "https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/maintenance/compaction.py"
click node_benchmarks "https://github.com/sxijyoti/incremental-lakehouse-cdc/tree/main/src/benchmarks"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer,node_kafka,node_consumer toneBlue
class node_spark,node_merge toneAmber
class node_icebergops,node_tables,node_catalog,node_storage,node_time toneMint
class node_maintenance,node_benchmarks toneRose
class node_sources,node_consumers toneIndigo
```

## Components

### CDC ingestion

- [Kafka producer](https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/ingestion/kafka_producer.py): `src/ingestion/kafka_producer.py`
- **Kafka topics**
- [Streaming consumer](https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/ingestion/streaming_consumer.py): `src/ingestion/streaming_consumer.py`

### Spark processing

- [Spark session](https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/lakehouse/spark_session.py): `src/lakehouse/spark_session.py`
- [CDC merge handler](https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/lakehouse/merge_handler.py): `src/lakehouse/merge_handler.py`

### Iceberg lakehouse

- [Table operations](https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/lakehouse/iceberg_operations.py): `src/lakehouse/iceberg_operations.py`
- **Iceberg tables**
- **Iceberg REST catalog**
- **MinIO storage**
- [Snapshot time travel](https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/lakehouse/time_travel.py): `src/lakehouse/time_travel.py`

### Lakehouse operations

- [Table maintenance](https://github.com/sxijyoti/incremental-lakehouse-cdc/blob/main/src/maintenance/compaction.py): `src/maintenance/compaction.py`
- [Performance benchmarks](https://github.com/sxijyoti/incremental-lakehouse-cdc/tree/main/src/benchmarks): `src/benchmarks`

### Other components

- **CDC sources**
- **Query consumers**

## Connections

- CDC sources → Kafka producer: generates events (evidence: `README.md`)
- Kafka producer → Kafka topics: publishes events (evidence: `src/ingestion/kafka_producer.py`)
- Kafka topics → Streaming consumer: streams events (evidence: `README.md`)
- Streaming consumer → CDC merge handler: delegates batches (evidence: `README.md`)
- Streaming consumer → Table operations: creates tables (evidence: `src/ingestion/streaming_consumer.py`)
- CDC merge handler → Iceberg tables: merges changes (evidence: `src/lakehouse/merge_handler.py`)
- Spark session → Iceberg REST catalog: configures connection (evidence: `src/lakehouse/spark_session.py`)
- Spark session → MinIO storage: configures access (evidence: `src/lakehouse/spark_session.py`)
- Table operations → Iceberg tables: creates and reads (evidence: `src/lakehouse/iceberg_operations.py`)
- Snapshot time travel → Iceberg tables: queries snapshots (evidence: `src/lakehouse/time_travel.py`)
- Snapshot time travel → Table operations: creates demo table (evidence: `src/lakehouse/time_travel.py`)
- Table maintenance → Iceberg tables: optimizes and cleans (evidence: `src/maintenance/compaction.py`)
- Iceberg tables → Query consumers: serves analytics (evidence: `README.md`)
