How hiepphde/cdc-streaming-pipeline works
Project type and purpose
• This is a real-time streaming data pipeline built on Apache Flink, Debezium, Kafka, and Postgres.
• Its purpose is to ingest financial transactions (from a Python generator), capture changes in Postgres via Debezium, stream them through Kafka, process/aggregate them in Flink, and write results back to Postgres.
File structure
Top-level artifacts:
• docker-compose.yml + Dockerfile & ojdbc8.jar: orchestration of Kafka, Zookeeper, Debezium, Control Center, Postgres, and Flink environment.
• TransactionGenerator/main.py: Python script that populates Postgres with transaction rows.
• exec-postgres-connector.properties: Debezium connector config for CDC.
• pom.xml + src/main/java/CdcPipeline: Maven-based Flink job (DataStreamJob.java) with DTOs and custom JSON deserializer.
• src/main/resources/log4j2.properties: logging config.
Patterns and roles:
– CDC pattern: Debezium monitors Postgres → writes change events to Kafka topics.
– Stream processing: Flink job consumes Kafka topics, applies transformations/aggregations, writes to Postgres sink via JDBC.
README insights
[ TransactionGenerator → Postgres → Debezium CDC → Kafka → Flink → Postgres ]
• Key components called out: Apache Flink, Debezium, Postgres, Kafka.
Detail level
• Subcomponents when relevant (e.g., Flink’s JobManager & TaskManagers).
• Configuration artifacts (exec-postgres-connector.properties feeding Debezium, Docker Compose controlling service lifecycles).
• Don’t overcrowd—focus on the five key stages of the pipeline and how they interconnect.