How rehaman24/aws-event-driven-incremental-etl works
Project Type and Purpose
• This is a serverless, event-driven ETL data pipeline running entirely on AWS.
• Purpose: Ingest daily flight CSVs into S3, automatically orchestrate metadata discovery and PySpark transformations in AWS Glue, load enriched results into an Amazon Redshift fact table, and notify via SNS.
Main Components
a. Ingestion:
• Amazon S3 Raw Bucket
• AWS EventBridge Rule (listens for new .csv uploads)
b. Orchestration & Metadata:
• AWS Step Functions State Machine
• Task: Start Glue Crawler
• Loop: Poll Crawler status until complete
• Task: Start Glue ETL Job (sync)
• Branch: Success or Failure → SNS
• AWS Glue Crawler & Glue Data Catalog (discovers schema)
c. Processing:
• AWS Glue ETL Job (PySpark)
• Reads incremental data via job bookmarking
• Reads Redshift dimension table (via Glue Catalog)
• Performs two-pass join to enrich origin and destination
• Writes to Redshift fact table
d. Storage & Serving:
• Amazon Redshift Cluster (dimension + fact tables)
e. Alerting:
• AWS SNS Topic for success / failure notifications
Relationships & Data Flow
• Within Step Functions:
Architectural Patterns & Principles
• Event-driven: Triggers flow on S3 events via EventBridge.
• Serverless: No servers to manage (Glue, Step Functions, EventBridge, SNS).
• Incremental Processing: Glue Job Bookmarking tracks processed partitions.
• Dependency Management: Step Functions polling ensures sequential tasks without race conditions.
• Two-pass join: Enriches data in two steps to resolve origin and destination metadata.