How aws-samples/aws-etl-orchestrator works
Project Type and Purpose
• This project is an ETL orchestration framework that uses AWS services (AWS Step Functions, AWS Lambda, AWS Glue, and Amazon Athena) to coordinate and execute ETL jobs.
• The README outlines an example where sales and marketing CSV datasets are processed, transformed (including aggregations and field renaming), and then joined into a final dataset. It also describes handling asynchronous job triggers (by file uploads to S3) and states monitoring via DynamoDB.
• Given this, the system is a serverless, cloud-native ETL orchestrator that leverages AWS CloudFormation for provisioning resources.
Analyzing the File Structure
• Top-Level Directories:
• “cloudformation”: Contains multiple CloudFormation templates that set up different AWS resource stacks (e.g., for the AWS Glue resources, Step Functions state machine, and Lambda functions – both gluerunner and athenarunner).
• “lambda”: Contains subdirectories for different AWS Lambda functions used as ETL runners. These include “athenarunner”, “gluerunner”, and “ons3objectcreated”. Each function has its configuration file along with setup files and requirements.
• “glue-scripts”: Contains the Python scripts which implement the actual ETL logic (processing/joining the sales and marketing datasets).
• “samples”: Provides sample CSV datasets used to trigger the ETL workflow.
• Additionally, the “build.py” script and Python Pynt tasks help package, deploy, and manage the stacks (create, update, delete).
Key Aspects from the README
• Detailed steps of the ETL workflow:
• Datasets for Sales and Marketing trigger wait states in the Step Functions state machine.
• AWS Lambda functions (termed “ETL Runners”) poll for activity tasks from the state machine.
• Once triggered, the Lambda functions invoke a corresponding ETL job (using AWS Glue or Amazon Athena).
• The Glue Runner Lambda also uses a DynamoDB table to track active jobs.
• CloudFormation templates are used to provision and configure AWS resources.
• The README provides example CLI commands and deployment instructions that confirm how the pieces are integrated.
• Clear labels for each major AWS component (Step Functions, Lambda (gluerunner/athenarunner/ons3objectcreated), Glue jobs, DynamoDB, S3 buckets, CloudFormation).
· The flow of triggers (e.g., S3 file uploads to Lambda).
· The initiation of activity tasks from Step Functions to the Lambda ETL Runners.
· Data flows between S3, AWS Glue, and DynamoDB.
· Control components (state machine, CloudFormation).
· Data processing components (ETL jobs in Glue/Athena).
· Data storage (S3 buckets, DynamoDB).
· Event sources/handlers (CloudWatch Events, S3 event-driven Lambda).
Final Assembly
• Start at the top with CloudFormation as the foundation, provisioning S3, Step Functions, Lambda functions, DynamoDB, and Glue resources.
• Next, detail the ETL orchestration:
• The Step Functions state machine acts as the command center.
• Parallel branches show different ETL jobs waiting for S3 uploads.
• ETL runners (Lambda functions) poll for tasks from Step Functions, start Glue jobs, and track state.
• Joined datasets and intermediary outputs saved to designated S3 prefixes.
• Error-handling and notifications (if any) via AWS CloudWatch log streams.