# mark-curran/snowflake-demo architecture

> Architecture diagram of the GitHub repository mark-curran/snowflake-demo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mark-curran/snowflake-demo
- Repository: https://github.com/mark-curran/snowflake-demo

## Overview

##### Project type and purpose
- This is an open-source data-ingestion demo showing two ingestion patterns into Snowflake:
- Bulk load of “customer” data via a Python service
- Streaming load of “order” data via Kafka/Snowpipe over Azure Event Hubs
- It also includes a Kafka‐consumer utility and Terraform scripts to provision Azure infra.

##### File structure
- Top-level:
- cli_setup.sh, .env, docker-compose.yml → orchestration & environment
- terraform/ → Azure infra (Event Hub, remote state)
- Services:
- load_via_python_api/ → Python bulk loader (bulk copy into Snowflake stage)
- load_via_kafka/ → Node.js/TypeScript Kafka producer + consumer + tests
- kafka_connector/ → Kafka Connect container with Snowpipe connector
- shared_schemas/ → JSON schemas + TypeScript package for validation

##### Main components
- Client/Producers
- Python Bulk Copier (bulk COPY INTO…)
- Node.js Kafka Producer (order events)
- Messaging / Streaming Infrastructure
- Azure Event Hub (Kafka‐compatible endpoint)
- Kafka Connect Snowpipe Connector container
- Optional Kafka Consumer utility (prints to stdout)
- Data Warehouse
- Snowflake Stage (for bulk files)
- Snowpipe Streaming channel
- Snowflake Tables (customer, order)
- Infra & Orchestration
- Terraform (provisions Event Hub, storage for remote state)
- Docker Compose (runs all containers)
- Shared Schemas (ensures consistent JSON schemas across services)

##### Relationships and data flows
- Bulk flow (Customer):
Python Bulk Copier → Snowflake Stage → Snowflake Table (customer)
- Streaming flow (Order):
Node.js Producer → Azure Event Hub → Kafka Connect → Snowpipe Channel → Snowflake Table (order)
- Auxiliary:
Node.js Consumer (in load_via_kafka) subscribes Event Hub and logs to stdout (no Snowflake)
- Provisioning:
Terraform → Azure resources, writes connection strings → .env & docker-compose

##### Architectural patterns and design principles
- Event-driven streaming (async, decoupled)
- Dual ingestion patterns: batch vs streaming
- Containerized, config-driven (Docker, .env)
- Infrastructure as code (Terraform for repeatable Azure setup)
- Shared schema validation for data consistency

##### Technologies, frameworks, libraries
- Python 3 (poetry/pyproject.toml)
- Node.js + TypeScript (jest tests, tsconfig)
- Azure Event Hubs (Kafka API)
- Confluent Kafka Connect Snowflake plugin
- Snowflake (COPY command, Snowpipe streaming)
- Terraform (HCL, remote state)
- Docker Compose

##### Detail-level checklist
- All containers and scripts (.sh, Dockerfiles) represented

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Infrastructure & Orchestration
    subgraph "Infrastructure & Orchestration"
        TF["Terraform Scripts"]:::infra
        DC["Docker Compose & .env"]:::infra
        RM[/"Azure Storage (Remote State)"/]:::cloud
        AEH[/"Azure Event Hub"/]:::cloud
        TF -->|"provisions Azure infra"| AEH
        TF -->|"sets up remote state"| RM
        DC -->|"orchestrates containers"| Clients
    end

    %% Shared Schemas
    subgraph "Shared Schemas"
        SS["shared_schemas"]:::shared
    end

    %% Clients / Producers / Utilities
    subgraph "Clients & Utilities"
        loadPy["Python Bulk Copier"]:::service
        loadKafkaProd["Node.js Kafka Producer"]:::service
        loadKafkaCons["Node.js Kafka Consumer"]:::service
        loadPy -->|uses schemas| SS
        loadKafkaProd -->|uses schemas| SS
        loadKafkaCons -->|uses schemas| SS
    end

    %% Bulk Load Pipeline
    loadPy -->|"bulk COPY INTO stage"| Stage[("Snowflake Stage")]:::db
    Stage -->|"COPY INTO customer table"| CustTable[("Snowflake Customer Table")]:::db

    %% Streaming Pipeline
    loadKafkaProd -->|"Kafka API (JSON)"| AEH
    AEH -->|"Kafka Topic"| KC["Kafka Connect Snowpipe Connector"]:::service
    KC -->|"Snowpipe Streaming"| SPChannel[("Snowpipe Channel")]:::db
    SPChannel -->|"INSERT into order table"| OrderTable[("Snowflake Order Table")]:::db

    %% Auxiliary Consumer
    loadKafkaCons -->|"Kafka API"| AEH

    %% Click Events
    click loadPy "https://github.com/mark-curran/snowflake-demo/tree/main/load_via_python_api/"
    click loadKafkaProd "https://github.com/mark-curran/snowflake-demo/tree/main/load_via_kafka/"
    click loadKafkaCons "https://github.com/mark-curran/snowflake-demo/blob/main/load_via_kafka/src/consume.ts"
    click KC "https://github.com/mark-curran/snowflake-demo/tree/main/kafka_connector/"
    click SS "https://github.com/mark-curran/snowflake-demo/tree/main/shared_schemas/"
    click TF "https://github.com/mark-curran/snowflake-demo/tree/main/terraform/"
    click DC "https://github.com/mark-curran/snowflake-demo/blob/main/docker-compose.yml"

    %% Styles
    classDef service fill:#AEDFF7,stroke:#3399FF,stroke-width:2px
    classDef cloud fill:#FFE0B2,stroke:#FF9900,stroke-width:2px,color:#663300
    classDef db fill:#E1F5A9,stroke:#6AA84F,stroke-width:2px
    classDef infra fill:#D9D2E9,stroke:#8E7CC3,stroke-width:2px
    classDef shared fill:#F4CCCC,stroke:#CC0000,stroke-width:2px
```
