How zacharydhamilton/realtime-datawarehousing works
Project Type and Purpose
• This is a real-time data-warehousing workshop/demo that uses Confluent Cloud to stream CDC events from two Postgres databases, enrich and join them with ksqlDB, then sink the enriched order stream into a data warehouse or analytics engine (Databricks, Snowflake, or Imply Polaris).
High-Level Architecture Components
a. Source Databases
• Postgres “customers” and Postgres “products” instances
• Can be run locally via Docker (Dockerfiles under postgres/...) or provisioned in AWS/GCP via Terraform (terraform/aws and terraform/gcp).
b. Confluent Cloud Platform
• Kafka Cluster (topics, partitions)
• Schema Registry (for AVRO schemas)
• Kafka Connect (Debezium CDC Source Connectors + Sink Connector)
• ksqlDB Cluster (streams, tables, joins)
c. Data Warehouse / Analytics Sink
• Databricks Delta Lake or Snowflake or Imply Polaris
• Polaris demo folder (/polaris/README.md)
Relationships and Data Flows
• ksqlDB uses topics as inputs, defines Streams & Tables, joins them to produce orders_enriched.
Architectural Patterns & Principles
• CDC ingestion with Debezium (event sourcing pattern)
• Stream processing with ksqlDB (table abstraction over streams + join)
• Infrastructure as code (Terraform) for DB provisioning
• Containerization for local demos (Dockerfiles + CSV seed data)
• Schema management via Schema Registry to enforce AVRO schemas consistently