How memiiso/debezium-server-iceberg works
Project type and purpose
• This is an open-source connector plugin for Debezium Server (“Debezium Iceberg Consumer”).
• Its purpose is to capture Change Data Capture (CDC) events from relational databases via Debezium and write them in real time into Apache Iceberg tables—without requiring Kafka, Spark, or a streaming platform.
File structure
• Top-level modules:
• debezium-server-iceberg-sink: the core Java library implementing the Debezium Server consumer SPI.
• debezium-server-iceberg-dist: assembly module that bundles the connector with a runner script, example config, Dockerfile, etc.
• python: lightweight Python runner for launching the packaged jar via CLI.
• Within the sink module:
• io.debezium.server.iceberg: main entrypoints (config classes, consumer implementations).
• Subpackages for converter, history, offset, storage, tableoperator, batchsizewait—each implementing a strategy or SPI interface.
• Configuration files:
• application.properties.example under dist module; uses MicroProfile Config.
• Build and CI: Maven pom.xml, Dockerfile, GitHub Actions workflows, .run for IntelliJ run configurations.
README for additional insights
• Describes usage: build with Maven, unzip distribution, configure application.properties, then run run.sh.
• Mentions append/upsert modes, no external stream platform required.
Main components and interactions
a. Components
1. Source Database (MySQL, PostgreSQL, MongoDB, etc.)
2. Debezium Engine (embedded in Debezium Server)
3. Iceberg Consumer Plugin
• Config Loader (MicroProfile Config)
• Event Converters (JSON, Struct)
• Schema History Store (IcebergSchemaHistory)
• Offset Store (IcebergOffsetBackingStore)
• Table Operators & Writers (PartitionedAppendWriter, Upsert/DeltaWriters)
• BatchSizeWait Strategies
4. Iceberg Catalog (HiveMetastore, REST catalog, JDBC catalog, S3)
5. Iceberg Table (data files + metadata)
6. Distribution Wrapper
• run.sh / Python runner
• Dockerfile
7. External Services
• Object storage (S3, HDFS)
• Metadata store (Hive, Glue)
b. Relationships / Data Flow
1. CDC events flow from Source DB → Debezium Engine.
2. Debezium Engine pushes events to the Iceberg Consumer Plugin.
3. Plugin:
• Loads configuration → initializes Catalog and TableOperator.
• For each event: passes through Converter → obtains structured record.
• Consults Schema History Store to evolve schema.
• Uses Offset Store to track progress.
• Applies Partitioning / Upsert logic via TableOperator → writes to Iceberg via Iceberg API.
• Optionally blocks via BatchSizeWait before flushing.
4. Iceberg API writes new data files and updates table metadata in the Catalog, storing data in object storage.
c. Architectural Patterns / Principles
• SPI & plugin model (Debezium Server consumer SPI)
• Strategy Pattern (Converter, TableOperator, BatchSizeWait)
• MicroProfile Config / CDI for pluggable configuration
• Modular layering: conversion, state management (offset/history), write layer
• Separation of concerns: convert vs state vs storage vs distribution
d. Key Technologies
• Java 21, Maven, Debezium Server SPI
• Apache Iceberg (Table API, Catalog API)
• MicroProfile Config, CDI (beans.xml)
• Python (for runner), Bash (run.sh), Docker
Detailed capture
• Break out sub-components in the sink:
• ConfigLoader → ConverterChain → SchemaHistory → OffsetStore → TableOperator → IcebergAPI
• Optional extensions (batch strategies, custom converters).
• External services: S3, Hive Metastore, JDBC catalog, etc.