# rehaman24/aws-event-driven-incremental-etl architecture

> Architecture diagram of the GitHub repository rehaman24/aws-event-driven-incremental-etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rehaman24/aws-event-driven-incremental-etl
- Repository: https://github.com/rehaman24/aws-event-driven-incremental-etl

## Overview

##### Project Type and Purpose
- This is a serverless, event-driven ETL data pipeline running entirely on AWS.
- Purpose: Ingest daily flight CSVs into S3, automatically orchestrate metadata discovery and PySpark transformations in AWS Glue, load enriched results into an Amazon Redshift fact table, and notify via SNS.

##### Main Components
a. Ingestion:
- Amazon S3 Raw Bucket
- AWS EventBridge Rule (listens for new .csv uploads)
b. Orchestration & Metadata:
- AWS Step Functions State Machine
- Task: Start Glue Crawler
- Loop: Poll Crawler status until complete
- Task: Start Glue ETL Job (sync)
- Branch: Success or Failure → SNS
- AWS Glue Crawler & Glue Data Catalog (discovers schema)
c. Processing:
- AWS Glue ETL Job (PySpark)
- Reads incremental data via job bookmarking
- Reads Redshift dimension table (via Glue Catalog)
- Performs two-pass join to enrich origin and destination
- Writes to Redshift fact table
d. Storage & Serving:
- Amazon Redshift Cluster (dimension + fact tables)
e. Alerting:
- AWS SNS Topic for success / failure notifications

##### Relationships & Data Flow
- Within Step Functions:

##### Architectural Patterns & Principles
- Event-driven: Triggers flow on S3 events via EventBridge.
- Serverless: No servers to manage (Glue, Step Functions, EventBridge, SNS).
- Incremental Processing: Glue Job Bookmarking tracks processed partitions.
- Dependency Management: Step Functions polling ensures sequential tasks without race conditions.
- Two-pass join: Enriches data in two steps to resolve origin and destination metadata.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Ingestion Layer
    subgraph "Ingestion"
        S3Raw[(Amazon S3 Raw Bucket)]:::storage
        EventBridge((EventBridge Rule)):::event
    end

    S3Raw -->|"S3:ObjectCreated"| EventBridge

    %% Orchestration Layer
    subgraph "Orchestration"
        StepFunctions{{Step Functions State Machine}}:::compute
        GlueCrawler[Glue Crawler]:::compute
        CrawlerChoice[[Poll Crawler Status]]:::compute
        GlueJob["Glue ETL Job\nBookmarking: processes only new files\nTwo-pass join"]:::compute
    end

    EventBridge -->|"Trigger"| StepFunctions
    StepFunctions -->|"Start Glue Crawler"| GlueCrawler
    GlueCrawler -->|"Crawler status"| CrawlerChoice
    CrawlerChoice -->|"Not complete"| GlueCrawler
    CrawlerChoice -->|"Complete"| StepFunctions
    StepFunctions -->|"startJobRun.sync"| GlueJob
    GlueJob -->|"Job status"| StepFunctions
    StepFunctions -->|"Success"| SNS
    StepFunctions -->|"Failure"| SNS

    %% Processing Layer
    subgraph "Processing"
        GlueJob
    end

    %% Storage Layer
    subgraph "Storage"
        Redshift[(Amazon Redshift Cluster)]:::storage
    end

    GlueJob -->|"Write to Redshift"| Redshift

    %% Alerting Layer
    subgraph "Alerting"
        SNS((SNS Topic)):::event
    end

    classDef compute fill:#cce5ff,stroke:#0366d6,color:#000000;
    classDef storage fill:#e2f0d9,stroke:#27632a,color:#000000;
    classDef event fill:#ffe5b4,stroke:#d97706,color:#000000;
```
