How sxijyoti/incremental-lakehouse-cdc works
Incremental Lakehouse CDC streams change events from Kafka through Spark Structured Streaming into transactional Iceberg tables. The consumer parses events and delegates micro-batch application to a merge handler, which deduplicates by ID and applies inserts, updates, and deletes. The project also exposes snapshot time travel, incremental reads, table maintenance, and benchmark utilities. Spark is the shared runtime/configuration layer for the catalog and object storage. Some documented capabilities, including benchmarks, have no sampled implementation excerpt, so their connections are not asserted.
Main entry points: streaming_consumer.py Streaming consumer · iceberg_operations.py Table operations · spark_session.py Spark session · time_travel.py Snapshot time travel
Components 14
CDC ingestion
- Kafka producer
src/ingestion/kafka_producer.py - Kafka topics
- Streaming consumer
src/ingestion/streaming_consumer.py
Spark processing
- Spark session
src/lakehouse/spark_session.py - CDC merge handler
src/lakehouse/merge_handler.py
Iceberg lakehouse
- Table operations
src/lakehouse/iceberg_operations.py - Iceberg tables
- Iceberg REST catalog
- MinIO storage
- Snapshot time travel
src/lakehouse/time_travel.py
Lakehouse operations
- Table maintenance
src/maintenance/compaction.py - Performance benchmarks
src/benchmarks
Other components
- CDC sources
- Query consumers
Connections 13
- CDC sources → Kafka producer: generates events
README.md - Kafka producer → Kafka topics: publishes events
src/ingestion/kafka_producer.py - Kafka topics → Streaming consumer: streams events
README.md - Streaming consumer → CDC merge handler: delegates batches
README.md - Streaming consumer → Table operations: creates tables
src/ingestion/streaming_consumer.py - CDC merge handler → Iceberg tables: merges changes
src/lakehouse/merge_handler.py - Spark session → Iceberg REST catalog: configures connection
src/lakehouse/spark_session.py - Spark session → MinIO storage: configures access
src/lakehouse/spark_session.py - Table operations → Iceberg tables: creates and reads
src/lakehouse/iceberg_operations.py - Snapshot time travel → Iceberg tables: queries snapshots
src/lakehouse/time_travel.py - Snapshot time travel → Table operations: creates demo table
src/lakehouse/time_travel.py - Table maintenance → Iceberg tables: optimizes and cleans
src/maintenance/compaction.py - Iceberg tables → Query consumers: serves analytics
README.md