# motazalratta/cdc-lakehouse-poc architecture

> Architecture diagram of the GitHub repository motazalratta/cdc-lakehouse-poc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/motazalratta/cdc-lakehouse-poc
- Repository: https://github.com/motazalratta/cdc-lakehouse-poc

## Overview

##### Project Type and Purpose
- This is a full-stack data-engineering PoC implementing a CDC (Change Data Capture) Lakehouse pipeline.
- Its goal is to ingest changes from a PostgreSQL source DB (via Debezium → Kafka), process and enrich streams in Flink SQL, store results in Iceberg tables on HDFS (with Hive Metastore), and enable ad-hoc analytics via Trino and Metabase.

##### Key Structural Elements from the File Tree
- Top-level “config” directory for service configs (HDFS, Hive Metastore, Kafka, Trino).
- “docker-compose.yml” orchestrates containerized services: Kafka (broker, schema-registry, connect, control-center), Flink (sql-client, jobmanager, taskmanager), HDFS (namenode, datanode), databases (Postgres, MySQL for metastore), Hive Metastore service, Trino (coordinator, worker), Metabase.
- “lib” holds JDBC and connector JARs for Flink and Kafka (Debezium plugin).

##### Insights from README
- Describes end-to-end flow: Postgres → Debezium → Kafka → Flink SQL → Iceberg on HDFS/Hive → Trino → Metabase.
- Lists exact versions and roles of each technology.

##### Final Checklist
- All 5 layers represented and labeled
- Configuration and lib dependencies noted as footnotes
- Emphasis on CDC → stream processing → lakehouse → BI path

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Ingestion Layer
    subgraph "Ingestion Layer"
        direction TB
        Postgres["PostgreSQL Source DB"]:::ingest
        DebeziumConnector["Debezium PostgreSQL Connector"]:::ingest
        DebeziumConfig["Debezium Config"]:::ingest
        KafkaBroker["Kafka Broker"]:::ingest
        SchemaRegistry["Schema Registry"]:::ingest
        ControlCenter["Kafka Control Center"]:::ingest
        KafkaConfig["Kafka Config Directory"]:::ingest
    end

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        FlinkSQLClient["Flink SQL Client"]:::compute
        JobManager["Flink JobManager"]:::compute
        TaskManager["Flink TaskManager"]:::compute
        FlinkLibs["Flink & Iceberg JARs"]:::compute
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        HDFSNameNode["HDFS NameNode"]:::storage
        HDFSDataNode["HDFS DataNode"]:::storage
        HDFSConfig["HDFS Config"]:::storage
        IcebergTables["Iceberg Tables"]:::storage
        HiveMetastoreService["Hive Metastore Service"]:::storage
        HiveMetastoreConfig["Hive Metastore Config"]:::storage
        HiveMySQLDriver["MySQL JDBC Driver"]:::storage
    end

    %% Query Layer
    subgraph "Query Layer"
        direction TB
        TrinoCoordinator["Trino Coordinator"]:::query
        TrinoWorker["Trino Worker"]:::query
        TrinoCatalog["Trino Catalog Config"]:::query
    end

    %% BI Layer
    subgraph "BI Layer"
        direction TB
        Metabase["Metabase"]:::bi
    end

    %% Orchestration and Docs
    Compose["docker-compose.yml"]:::orchestration
    Docs["Documentation (docs/)"]:::orchestration

    %% Data Flow
    Postgres -->|"CDC events"| DebeziumConnector
    DebeziumConnector -->|"Produce to topics"| KafkaBroker
    KafkaBroker --> SchemaRegistry
    KafkaBroker --> ControlCenter

    KafkaBroker -->|"Consume topics"| FlinkSQLClient
    FlinkSQLClient --> JobManager
    JobManager --> TaskManager
    TaskManager -->|"Write Parquet/Avro"| IcebergTables

    IcebergTables --> HDFSNameNode
    IcebergTables --> HDFSDataNode
    HDFSNameNode <--> HiveMetastoreService
    HDFSDataNode <--> HiveMetastoreService

    TrinoCoordinator -->|"Query Iceberg via Hive Metastore"| HiveMetastoreService
    TrinoWorker --> TrinoCoordinator
    HiveMetastoreService --> TrinoCatalog
    Metabase -->|"ODBC/JDBC"| TrinoCoordinator

    %% Orchestration connections
    Compose -.-> Postgres
    Compose -.-> DebeziumConnector
    Compose -.-> KafkaBroker
    Compose -.-> JobManager
    Compose -.-> HDFSNameNode
    Compose -.-> HiveMetastoreService
    Compose -.-> TrinoCoordinator
    Compose -.-> Metabase
    Docs -.-> Compose

    %% Click Events
    click DebeziumConnector "https://github.com/motazalratta/cdc-lakehouse-poc/tree/main/lib/kafka/connect-plugins/debezium-connector-postgres"
    click DebeziumConfig "https://github.com/motazalratta/cdc-lakehouse-poc/blob/main/config/kafka/debezium_postgres.json"
    click KafkaConfig "https://github.com/motazalratta/cdc-lakehouse-poc/tree/main/config/kafka"
    click FlinkLibs "https://github.com/motazalratta/cdc-lakehouse-poc/tree/main/lib/flink/common"
    click HDFSConfig "https://github.com/motazalratta/cdc-lakehouse-poc/blob/main/config/hdfs/hadoop.env"
    click HiveMetastoreConfig "https://github.com/motazalratta/cdc-lakehouse-poc/blob/main/config/hive-metastore/hive-site.xml"
    click HiveMySQLDriver "https://github.com/motazalratta/cdc-lakehouse-poc/blob/main/lib/jdbc/mysql-connector-j-8.0.31.jar"
    click TrinoCatalog "https://github.com/motazalratta/cdc-lakehouse-poc/blob/main/config/trino/catalog/iceberg.properties"
    click TrinoCoordinator "https://github.com/motazalratta/cdc-lakehouse-poc/blob/main/config/trino/coordinator/config.properties"
    click TrinoWorker "https://github.com/motazalratta/cdc-lakehouse-poc/blob/main/config/trino/worker/config.properties"
    click Compose "https://github.com/motazalratta/cdc-lakehouse-poc/blob/main/docker-compose.yml"
    click Docs "https://github.com/motazalratta/cdc-lakehouse-poc/tree/main/docs/"

    %% Styles
    classDef ingest fill:#AEDFF7,stroke:#1E90FF,color:#000
    classDef compute fill:#C8E6C9,stroke:#388E3C,color:#000
    classDef storage fill:#FFE0B2,stroke:#F57C00,color:#000
    classDef query fill:#E1BEE7,stroke:#8E24AA,color:#000
    classDef bi fill:#FFF59D,stroke:#FBC02D,color:#000
    classDef orchestration fill:#ECEFF1,stroke:#607D8B,color:#000
```
