# hiepphde/cdc-streaming-pipeline architecture

> Architecture diagram of the GitHub repository hiepphde/cdc-streaming-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hiepphde/cdc-streaming-pipeline
- Repository: https://github.com/hiepphde/cdc-streaming-pipeline

## Overview

##### Project type and purpose
- This is a real-time streaming data pipeline built on Apache Flink, Debezium, Kafka, and Postgres.
- Its purpose is to ingest financial transactions (from a Python generator), capture changes in Postgres via Debezium, stream them through Kafka, process/aggregate them in Flink, and write results back to Postgres.

##### File structure
Top-level artifacts:
- docker-compose.yml + Dockerfile & ojdbc8.jar: orchestration of Kafka, Zookeeper, Debezium, Control Center, Postgres, and Flink environment.
- TransactionGenerator/main.py: Python script that populates Postgres with transaction rows.
- exec-postgres-connector.properties: Debezium connector config for CDC.
- pom.xml + src/main/java/CdcPipeline: Maven-based Flink job (DataStreamJob.java) with DTOs and custom JSON deserializer.
- src/main/resources/log4j2.properties: logging config.
Patterns and roles:
– CDC pattern: Debezium monitors Postgres → writes change events to Kafka topics.
– Stream processing: Flink job consumes Kafka topics, applies transformations/aggregations, writes to Postgres sink via JDBC.

##### README insights
[ TransactionGenerator → Postgres → Debezium CDC → Kafka → Flink → Postgres ]
- Key components called out: Apache Flink, Debezium, Postgres, Kafka.

##### Detail level
- Subcomponents when relevant (e.g., Flink’s JobManager & TaskManagers).
- Configuration artifacts (exec-postgres-connector.properties feeding Debezium, Docker Compose controlling service lifecycles).
- Don’t overcrowd—focus on the five key stages of the pipeline and how they interconnect.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Source Generation
    TransactionGenerator["TransactionGenerator\n(Python Script)"]:::source
    TransactionGenerator -->|"JDBC INSERT"| PostgresSource[(Postgres OLTP)]:::source

    %% Docker Compose Group
    subgraph "Docker Compose"
        DockerCompose["docker-compose.yml"]:::config
        Zookeeper["Zookeeper"]:::infra
        ControlCenter["Kafka Control Center"]:::infra
        PostgresSource
        Debezium["Debezium Connector"]:::infra
        Kafka[(Kafka Cluster)]:::infra
    end

    %% Debezium CDC Flow
    PostgresSource -->|"CDC Event"| Debezium
    Debezium -->|"Publish to Topic"| Kafka

    %% Processing Layer
    Kafka -->|"Consume Topics"| FlinkCluster
    subgraph "Flink Cluster"
        direction TB
        JobManager["JobManager"]:::processor
        TaskManagers["TaskManagers"]:::processor
        DataStreamJob["DataStreamJob\n(CdcPipeline)"]:::processor
    end
    Kafka -->|"Flink DataStream"| DataStreamJob

    %% Sink
    DataStreamJob -->|"JDBC Sink"| PostgresSink[(Postgres Aggregated Store)]:::sink

    %% Configuration and Artifacts
    DebeziumConfig["exec-postgres-connector.properties"]:::config
    Pom["pom.xml"]:::config
    FlinkDockerfile["Dockerfile"]:::config
    JDBCDriver["ojdbc8.jar"]:::config
    JSONSchema["JSONValueDeserializationSchema.java"]:::config
    TransactionDTO["Transaction.java"]:::config
    LogConfig["log4j2.properties"]:::config
    README["README.md"]:::config

    %% Artifact Links
    Kafka --> DebeziumConfig
    FlinkCluster --> Pom
    FlinkCluster --> JSONSchema
    FlinkCluster --> TransactionDTO
    FlinkCluster --> LogConfig
    FlinkCluster --> FlinkDockerfile
    FlinkCluster --> JDBCDriver
    TransactionGenerator --> README

    %% Click Events
    click TransactionGenerator "https://github.com/hiepphde/cdc-streaming-pipeline/blob/main/TransactionGenerator/main.py"
    click DebeziumConfig "https://github.com/hiepphde/cdc-streaming-pipeline/blob/main/exec-postgres-connector.properties"
    click DockerCompose "https://github.com/hiepphde/cdc-streaming-pipeline/blob/main/docker-compose.yml"
    click FlinkDockerfile "https://github.com/hiepphde/cdc-streaming-pipeline/tree/main/Dockerfile"
    click JDBCDriver "https://github.com/hiepphde/cdc-streaming-pipeline/blob/main/Dockerfile/ojdbc8.jar"
    click Pom "https://github.com/hiepphde/cdc-streaming-pipeline/blob/main/pom.xml"
    click DataStreamJob "https://github.com/hiepphde/cdc-streaming-pipeline/blob/main/src/main/java/CdcPipeline/DataStreamJob.java"
    click JSONSchema "https://github.com/hiepphde/cdc-streaming-pipeline/blob/main/src/main/java/Deserializer/JSONValueDeserializationSchema.java"
    click TransactionDTO "https://github.com/hiepphde/cdc-streaming-pipeline/blob/main/src/main/java/Dto/Transaction.java"
    click LogConfig "https://github.com/hiepphde/cdc-streaming-pipeline/blob/main/src/main/resources/log4j2.properties"
    click README "https://github.com/hiepphde/cdc-streaming-pipeline/blob/main/README.md"

    %% Styles
    classDef source fill:#add8e6,stroke:#000
    classDef infra fill:#ffa500,stroke:#000
    classDef processor fill:#90ee90,stroke:#000
    classDef sink fill:#dda0dd,stroke:#000
    classDef config fill:#f0e68c,stroke:#000
```
