How mark-curran/snowflake-demo works
Project type and purpose
• This is an open-source data-ingestion demo showing two ingestion patterns into Snowflake:
• Bulk load of “customer” data via a Python service
• Streaming load of “order” data via Kafka/Snowpipe over Azure Event Hubs
• It also includes a Kafka‐consumer utility and Terraform scripts to provision Azure infra.
File structure
• Top-level:
• cli_setup.sh, .env, docker-compose.yml → orchestration & environment
• terraform/ → Azure infra (Event Hub, remote state)
• Services:
• load_via_python_api/ → Python bulk loader (bulk copy into Snowflake stage)
• load_via_kafka/ → Node.js/TypeScript Kafka producer + consumer + tests
• kafka_connector/ → Kafka Connect container with Snowpipe connector
• shared_schemas/ → JSON schemas + TypeScript package for validation
Main components
• Client/Producers
• Python Bulk Copier (bulk COPY INTO…)
• Node.js Kafka Producer (order events)
• Messaging / Streaming Infrastructure
• Azure Event Hub (Kafka‐compatible endpoint)
• Kafka Connect Snowpipe Connector container
• Optional Kafka Consumer utility (prints to stdout)
• Data Warehouse
• Snowflake Stage (for bulk files)
• Snowpipe Streaming channel
• Snowflake Tables (customer, order)
• Infra & Orchestration
• Terraform (provisions Event Hub, storage for remote state)
• Docker Compose (runs all containers)
• Shared Schemas (ensures consistent JSON schemas across services)
Relationships and data flows
• Bulk flow (Customer):
Python Bulk Copier → Snowflake Stage → Snowflake Table (customer)
• Streaming flow (Order):
Node.js Producer → Azure Event Hub → Kafka Connect → Snowpipe Channel → Snowflake Table (order)
• Auxiliary:
Node.js Consumer (in load_via_kafka) subscribes Event Hub and logs to stdout (no Snowflake)
• Provisioning:
Terraform → Azure resources, writes connection strings → .env & docker-compose
Architectural patterns and design principles
• Event-driven streaming (async, decoupled)
• Dual ingestion patterns: batch vs streaming
• Containerized, config-driven (Docker, .env)
• Infrastructure as code (Terraform for repeatable Azure setup)
• Shared schema validation for data consistency
Technologies, frameworks, libraries
• Python 3 (poetry/pyproject.toml)
• Node.js + TypeScript (jest tests, tsconfig)
• Azure Event Hubs (Kafka API)
• Confluent Kafka Connect Snowflake plugin
• Snowflake (COPY command, Snowpipe streaming)
• Terraform (HCL, remote state)
• Docker Compose
Detail-level checklist
• All containers and scripts (.sh, Dockerfiles) represented