# kushalkhadka7/streaming-cdc architecture

> Architecture diagram of the GitHub repository kushalkhadka7/streaming-cdc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kushalkhadka7/streaming-cdc
- Repository: https://github.com/kushalkhadka7/streaming-cdc

## Overview

This repository is described as a PostgreSQL/MSSQL change-data-capture pipeline: producers write database changes, Debezium publishes table-specific events to Kafka, and Flink processes them for Kafka or Elasticsearch sinks. The README also names the schema registry, Kafka Control Center, and product/order load generators. The supplied tree contains only installed Python packages, not the application implementation, so paths are null and edges reflect documented flow rather than inspected calls.

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sources["Data Sources"]
  node_product_loader["Product Loader"]
  node_order_loader["Order Loader"]
  node_postgres[("PostgreSQL")]
  node_mssql[("MSSQL")]
end

subgraph group_capture["Change Capture"]
  node_debezium["Debezium"]
  node_kafka["Kafka Cluster"]
  node_table_topics["Table Topics"]
  node_schema_registry["Schema Registry"]
end

subgraph group_processing["Stream Processing"]
  node_flink["Flink Cluster"]
  node_flink_job["Flink Job"]
  node_stream_sink["Kafka Sink"]
  node_elasticsearch[("Elasticsearch Sink")]
end

subgraph group_operations["Operations"]
  node_control_center["Kafka Control Center"]
  node_flink_dashboard["Flink Dashboard"]
end

node_services(("Writing Services"))

node_services -->|"writes changes"| node_postgres
node_services -->|"writes changes"| node_mssql
node_product_loader -->|"loads products"| node_mssql
node_order_loader -->|"writes orders"| node_postgres
node_postgres -->|"captures changes"| node_debezium
node_mssql -->|"captures changes"| node_debezium
node_debezium -->|"publishes events"| node_table_topics
node_table_topics -->|"stored in"| node_kafka
node_debezium -.->|"uses schemas"| node_schema_registry
node_flink_job -->|"reads streams"| node_kafka
node_flink -->|"runs job"| node_flink_job
node_flink_job -->|"sends processed data"| node_stream_sink
node_flink_job -->|"writes processed data"| node_elasticsearch
node_control_center -->|"monitors cluster"| node_kafka
node_flink_dashboard -->|"shows status"| node_flink

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_product_loader,node_order_loader,node_postgres,node_mssql toneBlue
class node_debezium,node_kafka,node_table_topics,node_schema_registry toneAmber
class node_flink,node_flink_job,node_stream_sink,node_elasticsearch toneMint
class node_control_center,node_flink_dashboard toneRose
class node_services toneIndigo
```

## Components

### Data Sources

- **Product Loader**
- **Order Loader**
- **PostgreSQL**
- **MSSQL**

### Change Capture

- **Debezium**
- **Kafka Cluster**
- **Table Topics**
- **Schema Registry**

### Stream Processing

- **Flink Cluster**
- **Flink Job**
- **Kafka Sink**
- **Elasticsearch Sink**

### Operations

- **Kafka Control Center**
- **Flink Dashboard**

### Other components

- **Writing Services**

## Connections

- Writing Services → PostgreSQL: writes changes
- Writing Services → MSSQL: writes changes
- Product Loader → MSSQL: loads products
- Order Loader → PostgreSQL: writes orders
- PostgreSQL → Debezium: captures changes
- MSSQL → Debezium: captures changes
- Debezium → Table Topics: publishes events
- Table Topics → Kafka Cluster: stored in
- Debezium → Schema Registry: uses schemas
- Flink Job → Kafka Cluster: reads streams
- Flink Cluster → Flink Job: runs job
- Flink Job → Kafka Sink: sends processed data
- Flink Job → Elasticsearch Sink: writes processed data
- Kafka Control Center → Kafka Cluster: monitors cluster
- Flink Dashboard → Flink Cluster: shows status