# memiiso/debezium-server-iceberg architecture

> Architecture diagram of the GitHub repository memiiso/debezium-server-iceberg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/memiiso/debezium-server-iceberg
- Repository: https://github.com/memiiso/debezium-server-iceberg

## Overview

##### Project type and purpose
- This is an open-source connector plugin for Debezium Server (“Debezium Iceberg Consumer”).
- Its purpose is to capture Change Data Capture (CDC) events from relational databases via Debezium and write them in real time into Apache Iceberg tables—without requiring Kafka, Spark, or a streaming platform.

##### File structure
- Top-level modules:
- `debezium-server-iceberg-sink`: the core Java library implementing the Debezium Server consumer SPI.
- `debezium-server-iceberg-dist`: assembly module that bundles the connector with a runner script, example config, Dockerfile, etc.
- `python`: lightweight Python runner for launching the packaged jar via CLI.
- Within the sink module:
- `io.debezium.server.iceberg`: main entrypoints (config classes, consumer implementations).
- Subpackages for converter, history, offset, storage, tableoperator, batchsizewait—each implementing a strategy or SPI interface.
- Configuration files:
- `application.properties.example` under dist module; uses MicroProfile Config.
- Build and CI: Maven `pom.xml`, `Dockerfile`, GitHub Actions workflows, `.run` for IntelliJ run configurations.

##### README for additional insights
- Describes usage: build with Maven, unzip distribution, configure `application.properties`, then run `run.sh`.
- Mentions append/upsert modes, no external stream platform required.

##### Main components and interactions
a. Components
1. Source Database (MySQL, PostgreSQL, MongoDB, etc.)
2. Debezium Engine (embedded in Debezium Server)
3. Iceberg Consumer Plugin
- Config Loader (MicroProfile Config)
- Event Converters (JSON, Struct)
- Schema History Store (IcebergSchemaHistory)
- Offset Store (IcebergOffsetBackingStore)
- Table Operators & Writers (PartitionedAppendWriter, Upsert/DeltaWriters)
- BatchSizeWait Strategies
4. Iceberg Catalog (HiveMetastore, REST catalog, JDBC catalog, S3)
5. Iceberg Table (data files + metadata)
6. Distribution Wrapper
- run.sh / Python runner
- Dockerfile
7. External Services
- Object storage (S3, HDFS)
- Metadata store (Hive, Glue)

b. Relationships / Data Flow
1. CDC events flow from Source DB → Debezium Engine.
2. Debezium Engine pushes events to the Iceberg Consumer Plugin.
3. Plugin:
- Loads configuration → initializes Catalog and TableOperator.
- For each event: passes through Converter → obtains structured record.
- Consults Schema History Store to evolve schema.
- Uses Offset Store to track progress.
- Applies Partitioning / Upsert logic via TableOperator → writes to Iceberg via Iceberg API.
- Optionally blocks via BatchSizeWait before flushing.
4. Iceberg API writes new data files and updates table metadata in the Catalog, storing data in object storage.

c. Architectural Patterns / Principles
- SPI & plugin model (Debezium Server consumer SPI)
- Strategy Pattern (Converter, TableOperator, BatchSizeWait)
- MicroProfile Config / CDI for pluggable configuration
- Modular layering: conversion, state management (offset/history), write layer
- Separation of concerns: convert vs state vs storage vs distribution

d. Key Technologies
- Java 21, Maven, Debezium Server SPI
- Apache Iceberg (Table API, Catalog API)
- MicroProfile Config, CDI (beans.xml)
- Python (for runner), Bash (run.sh), Docker

##### Detailed capture
- Break out sub-components in the sink:
- ConfigLoader → ConverterChain → SchemaHistory → OffsetStore → TableOperator → IcebergAPI
- Optional extensions (batch strategies, custom converters).
- External services: S3, Hive Metastore, JDBC catalog, etc.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Source
    DB["Source Database"]:::external

    %% Debezium Engine
    Debezium["Debezium Engine"]:::core

    %% SPI Entrypoints
    subgraph "SPI Entrypoints"
        direction TB
        ICE["IcebergChangeConsumer"]:::core
        IEC["IcebergEventsChangeConsumer"]:::core
    end

    %% Core Plugin Pipeline
    subgraph "Iceberg Consumer Plugin"
        direction TB
        ConfigLoader["Config Loader"]:::core
        subgraph "Event Converters"
            direction TB
            JSONConv["JsonEventConverter"]:::core
            StructConv["StructEventConverter"]:::core
        end
        SchemaHistory["IcebergSchemaHistory"]:::core
        OffsetStore["IcebergOffsetBackingStore"]:::core
        subgraph "Table Operators & Writers"
            direction TB
            AppendWriter["PartitionedAppendWriter"]:::core
            DeltaWriter["PartitionedDeltaWriter"]:::core
            UnpartWriter["UnpartitionedDeltaWriter"]:::core
        end
        subgraph "BatchSizeWait Strategies"
            direction TB
            MaxWait["MaxBatchSizeWait"]:::core
            NoWait["NoBatchSizeWait"]:::core
        end
    end

    %% Iceberg API & Storage
    IcebergAPI["Iceberg API"]:::iceberg
    subgraph "Iceberg Storage"
        direction TB
        Catalog["Iceberg Catalog"]:::iceberg
        Table["Iceberg Table"]:::iceberg
        Storage["Object Storage"]:::storage
    end

    %% Distribution
    subgraph "Distribution Wrapper"
        direction TB
        RunScript["run.sh / Dockerfile"]:::dist
        PythonRunner["Python Runner"]:::dist
    end

    %% Data Flows
    DB -->|"CDC events"| Debezium
    Debezium --> ICE
    Debezium --> IEC
    ICE --> ConfigLoader
    IEC --> ConfigLoader
    ConfigLoader --> JSONConv
    ConfigLoader --> StructConv
    JSONConv --> SchemaHistory
    StructConv --> SchemaHistory
    SchemaHistory --> OffsetStore
    OffsetStore --> AppendWriter
    OffsetStore --> DeltaWriter
    OffsetStore --> UnpartWriter
    AppendWriter --> MaxWait
    DeltaWriter --> MaxWait
    UnpartWriter --> NoWait
    MaxWait --> IcebergAPI
    NoWait --> IcebergAPI
    IcebergAPI --> Catalog
    Catalog --> Table
    Table --> Storage
    RunScript --> Debezium
    PythonRunner --> Debezium

    %% Click Events
    click ICE "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/IcebergChangeConsumer.java"
    click IEC "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/IcebergEventsChangeConsumer.java"
    click ConfigLoader "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/DebeziumConfig.java"
    click JSONConv "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/converter/JsonEventConverter.java"
    click StructConv "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/converter/StructEventConverter.java"
    click SchemaHistory "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/history/IcebergSchemaHistory.java"
    click OffsetStore "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/offset/IcebergOffsetBackingStore.java"
    click AppendWriter "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/tableoperator/PartitionedAppendWriter.java"
    click DeltaWriter "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/tableoperator/PartitionedDeltaWriter.java"
    click UnpartWriter "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/tableoperator/UnpartitionedDeltaWriter.java"
    click MaxWait "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/batchsizewait/MaxBatchSizeWait.java"
    click NoWait "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/batchsizewait/NoBatchSizeWait.java"
    click RunScript "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-dist/src/main/resources/distro/run.sh"
    click PythonRunner "https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-dist/src/main/resources/distro/debezium.py"
    click Storage "https://github.com/memiiso/debezium-server-iceberg/blob/master/examples/docker-compose.yml"

    %% Styles
    classDef core fill:#D6EAF8,stroke:#1F618D,stroke-width:1px;
    classDef iceberg fill:#D5F5E3,stroke:#27AE60,stroke-width:1px;
    classDef storage fill:#FCF3CF,stroke:#F1C40F,stroke-width:1px;
    classDef dist fill:#E8E8E8,stroke:#888888,stroke-width:1px;
    classDef external fill:#FDEDEC,stroke:#C0392B,stroke-width:1px;
```
