How motazalratta/cdc-lakehouse-poc works
Project Type and Purpose
• This is a full-stack data-engineering PoC implementing a CDC (Change Data Capture) Lakehouse pipeline.
• Its goal is to ingest changes from a PostgreSQL source DB (via Debezium → Kafka), process and enrich streams in Flink SQL, store results in Iceberg tables on HDFS (with Hive Metastore), and enable ad-hoc analytics via Trino and Metabase.
Key Structural Elements from the File Tree
• Top-level “config” directory for service configs (HDFS, Hive Metastore, Kafka, Trino).
• “docker-compose.yml” orchestrates containerized services: Kafka (broker, schema-registry, connect, control-center), Flink (sql-client, jobmanager, taskmanager), HDFS (namenode, datanode), databases (Postgres, MySQL for metastore), Hive Metastore service, Trino (coordinator, worker), Metabase.
• “lib” holds JDBC and connector JARs for Flink and Kafka (Debezium plugin).
Insights from README
• Describes end-to-end flow: Postgres → Debezium → Kafka → Flink SQL → Iceberg on HDFS/Hive → Trino → Metabase.
• Lists exact versions and roles of each technology.
Final Checklist
• All 5 layers represented and labeled
• Configuration and lib dependencies noted as footnotes
• Emphasis on CDC → stream processing → lakehouse → BI path