# srujangowda-10/aws-to-snowflake-streaming-data-pipeline architecture

> Architecture diagram of the GitHub repository srujangowda-10/aws-to-snowflake-streaming-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/srujangowda-10/aws-to-snowflake-streaming-data-pipeline
- Repository: https://github.com/srujangowda-10/aws-to-snowflake-streaming-data-pipeline

## Overview

##### Main Components

a) AWS Services Layer:
- Lambda Function (Data Generation)
- S3 Buckets (Raw and Transformed Data)
- AWS Glue (Data Transformation)

b) Snowflake Layer:
- External Stage
- Snowpipe
- Database Tables (Stage, SCD1, SCD2)
- Streams
- Tasks
- Stored Procedures

##### Connection Types
- Double lines: Streaming connections

##### Detailed Component Relationships

Start (Top):
**AWS Lambda (Python Faker)**
- ↓
- S3 (Raw Data)
- ↓
**AWS Glue (PySpark)**
- ↓
- S3 (Transformed Data)

Middle:
**Snowflake External Stage ← → Snowpipe**
- ↓
- Stage Table
- ↓
- Streams

Bottom:
**Stored Procedures**
- ↙     ↘
**SCD1 Table  SCD2 Table**

##### Important Elements to Highlight
- Automated data generation process
- Data transformation workflow
- SCD management mechanism
- Real-time streaming components
- Data quality checks and validation points

##### Technical Details
- File formats at each stage
- Key transformation steps
- Table structure indicators
- Stream processing logic
- Task scheduling information

Remember to:
- Use consistent sizing for similar components
- Clear labels for all components
- Both success and error paths
- Monitoring points
- Document all automation processes

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and classes
    classDef aws fill:#FF9900,stroke:#232F3E,color:white
    classDef storage fill:#4F5B93,stroke:#232F3E,color:white
    classDef snowflake fill:#29B5E8,stroke:#232F3E,color:white
    classDef process fill:#8A2BE2,stroke:#232F3E,color:white
    classDef dataflow fill:#98FB98,stroke:#232F3E
    
    %% AWS Layer
    subgraph AWS
        Lambda["AWS Lambda\n(Python Faker)"]:::aws
        S3Raw["S3 Raw Data\nBucket"]:::storage
        Glue["AWS Glue\n(PySpark)"]:::aws
        S3Trans["S3 Transformed\nData Bucket"]:::storage
    end

    %% Data Transformation Layer
    subgraph Transform
        TransformNotebook["Data Transformation\nNotebook"]:::process
        TransformScript["Transformation\nPySpark Script"]:::process
    end
    click TransformNotebook "https://github.com/SrujanGowda-10/AWS-to-Snowflake-Streaming-Data-Pipeline/blob/main/Data Transformation Notebok.ipynb"
    click TransformScript "https://github.com/SrujanGowda-10/AWS-to-Snowflake-Streaming-Data-Pipeline/blob/main/transformation job py script.txt"

    %% Snowflake Layer
    subgraph Snowflake
        ExtStage["External Stage"]:::snowflake
        Snowpipe["Snowpipe"]:::snowflake
        StageTable["Stage Table"]:::storage
        DataStream["Data Stream"]:::process
        Tasks["Tasks"]:::process
        StoredProc["Stored Procedures"]:::process
        SCD1["SCD1 Table"]:::storage
        SCD2["SCD2 Table"]:::storage
    end
    click ExtStage "https://github.com/SrujanGowda-10/AWS-to-Snowflake-Streaming-Data-Pipeline/blob/main/Script for creating external stage.txt"
    click Snowpipe "https://github.com/SrujanGowda-10/AWS-to-Snowflake-Streaming-Data-Pipeline/blob/main/Snowpipe-streams-tasks.txt"
    click DataStream "https://github.com/SrujanGowda-10/AWS-to-Snowflake-Streaming-Data-Pipeline/blob/main/Snowpipe-streams-tasks.txt"
    click Tasks "https://github.com/SrujanGowda-10/AWS-to-Snowflake-Streaming-Data-Pipeline/blob/main/Snowpipe-streams-tasks.txt"
    click StoredProc "https://github.com/SrujanGowda-10/AWS-to-Snowflake-Streaming-Data-Pipeline/blob/main/SQL Procedures.txt"
    click SCD1 "https://github.com/SrujanGowda-10/AWS-to-Snowflake-Streaming-Data-Pipeline/blob/main/Tables creation.txt"
    click SCD2 "https://github.com/SrujanGowda-10/AWS-to-Snowflake-Streaming-Data-Pipeline/blob/main/Tables creation.txt"
    click StageTable "https://github.com/SrujanGowda-10/AWS-to-Snowflake-Streaming-Data-Pipeline/blob/main/Tables creation.txt"

    %% Connections
    Lambda -->|Generate Data| S3Raw
    S3Raw -->|Raw Data| Glue
    Glue -->|Transform| S3Trans
    S3Trans -->|Load| ExtStage
    ExtStage -->|Auto-Ingest| Snowpipe
    Snowpipe -->|Copy Into| StageTable
    StageTable -->|Change Data| DataStream
    DataStream -->|Trigger| Tasks
    Tasks -->|Execute| StoredProc
    StoredProc -->|Update| SCD1
    StoredProc -->|Update| SCD2

    %% Connect transformation components
    Glue --- TransformNotebook
    Glue --- TransformScript

    %% Legend
    subgraph Legend
        L1[AWS Services]:::aws
        L2[Storage Components]:::storage
        L3[Snowflake Components]:::snowflake
        L4[Processing Steps]:::process
        L5[Data Flow]:::dataflow
    end
```
