# zacharydhamilton/realtime-datawarehousing architecture

> Architecture diagram of the GitHub repository zacharydhamilton/realtime-datawarehousing, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zacharydhamilton/realtime-datawarehousing
- Repository: https://github.com/zacharydhamilton/realtime-datawarehousing

## Overview

##### Project Type and Purpose
- This is a real-time data-warehousing workshop/demo that uses Confluent Cloud to stream CDC events from two Postgres databases, enrich and join them with ksqlDB, then sink the enriched order stream into a data warehouse or analytics engine (Databricks, Snowflake, or Imply Polaris).

##### High-Level Architecture Components
a. Source Databases
- Postgres “customers” and Postgres “products” instances
- Can be run locally via Docker (Dockerfiles under `postgres/...`) or provisioned in AWS/GCP via Terraform (`terraform/aws` and `terraform/gcp`).
b. Confluent Cloud Platform
- Kafka Cluster (topics, partitions)
- Schema Registry (for AVRO schemas)
- Kafka Connect (Debezium CDC Source Connectors + Sink Connector)
- ksqlDB Cluster (streams, tables, joins)
c. Data Warehouse / Analytics Sink
- Databricks Delta Lake or Snowflake or Imply Polaris
- Polaris demo folder (`/polaris/README.md`)

##### Relationships and Data Flows
- ksqlDB uses topics as inputs, defines Streams & Tables, joins them to produce `orders_enriched`.

##### Architectural Patterns & Principles
- CDC ingestion with Debezium (event sourcing pattern)
- Stream processing with ksqlDB (table abstraction over streams + join)
- Infrastructure as code (Terraform) for DB provisioning
- Containerization for local demos (Dockerfiles + CSV seed data)
- Schema management via Schema Registry to enforce AVRO schemas consistently

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Local / Cloud Infra
    subgraph "Local / Cloud Infra"
        direction TB
        subgraph "Dockerized Postgres" 
            direction TB
            PGC["Postgres: customers (Docker)"]:::source
            PGP["Postgres: products (Docker)"]:::source
        end
        subgraph "Terraform Provisioned Postgres"
            direction TB
            TF_AWS["Terraform AWS"]:::infra
            TF_GCP["Terraform GCP"]:::infra
        end
    end

    %% Confluent Cloud
    subgraph "Confluent Cloud"
        direction TB
        KafkaCluster["Kafka Cluster"]:::confluent
        SchemaRegistry["Schema Registry"]:::confluent
        subgraph "Kafka Connect"
            direction TB
            CDC_Cust["Debezium CDC Source: customers"]:::confluent
            CDC_Prod["Debezium CDC Source: products"]:::confluent
            SinkConn["Sink Connector"]:::confluent
        end
        subgraph "ksqlDB"
            direction TB
            StreamCust["ksqlDB Stream: customers_flattened"]:::confluent
            StreamProd["ksqlDB Stream: products_flattened"]:::confluent
            JoinEnrich["ksqlDB Table Join → orders_enriched"]:::confluent
        end
    end

    %% Analytics / Visualization
    subgraph "Analytics / Visualization"
        direction TB
        DW["Data Warehouse\n(Databricks / Snowflake)"]:::sink
        Polaris["Imply Polaris UI"]:::sink
    end

    %% Project Documentation
    ProjectDoc["README"]:::doc

    %% Data Flows
    PGC -->|AVRO over Kafka| CDC_Cust
    PGP -->|AVRO over Kafka| CDC_Prod
    CDC_Cust -->|produces topics| KafkaCluster
    CDC_Prod -->|produces topics| KafkaCluster
    KafkaCluster -->|"Connect & ksqlDB"| SchemaRegistry
    KafkaCluster -->|AVRO topics| StreamCust
    KafkaCluster -->|AVRO topics| StreamProd
    StreamCust -->|JSON re-keying| JoinEnrich
    StreamProd -->|JSON re-keying| JoinEnrich
    JoinEnrich -->|enriched topic| SinkConn
    SinkConn -->|AVRO over Kafka| DW
    JoinEnrich -->|AVRO over Kafka| Polaris

    %% Click events
    click PGC "https://github.com/zacharydhamilton/realtime-datawarehousing/tree/main/postgres/customers"
    click PGP "https://github.com/zacharydhamilton/realtime-datawarehousing/tree/main/postgres/products"
    click TF_AWS "https://github.com/zacharydhamilton/realtime-datawarehousing/tree/main/terraform/aws"
    click TF_GCP "https://github.com/zacharydhamilton/realtime-datawarehousing/tree/main/terraform/gcp"
    click Polaris "https://github.com/zacharydhamilton/realtime-datawarehousing/blob/main/polaris/README.md"
    click ProjectDoc "https://github.com/zacharydhamilton/realtime-datawarehousing/blob/main/README.md"

    %% Styles
    classDef source fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef confluent fill:#FFA500,stroke:#000,stroke-width:1px
    classDef sink fill:#90EE90,stroke:#000,stroke-width:1px
    classDef infra fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef doc fill:#FFFFFF,stroke:#000,stroke-width:1px,stroke-dasharray: 5 5
```
