# aws-samples/aws-etl-orchestrator architecture

> Architecture diagram of the GitHub repository aws-samples/aws-etl-orchestrator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws-samples/aws-etl-orchestrator
- Repository: https://github.com/aws-samples/aws-etl-orchestrator

## Overview

##### Project Type and Purpose
- This project is an ETL orchestration framework that uses AWS services (AWS Step Functions, AWS Lambda, AWS Glue, and Amazon Athena) to coordinate and execute ETL jobs.
- The README outlines an example where sales and marketing CSV datasets are processed, transformed (including aggregations and field renaming), and then joined into a final dataset. It also describes handling asynchronous job triggers (by file uploads to S3) and states monitoring via DynamoDB.
- Given this, the system is a serverless, cloud-native ETL orchestrator that leverages AWS CloudFormation for provisioning resources.

##### Analyzing the File Structure
- Top-Level Directories:
- “cloudformation”: Contains multiple CloudFormation templates that set up different AWS resource stacks (e.g., for the AWS Glue resources, Step Functions state machine, and Lambda functions – both gluerunner and athenarunner).
- “lambda”: Contains subdirectories for different AWS Lambda functions used as ETL runners. These include “athenarunner”, “gluerunner”, and “ons3objectcreated”. Each function has its configuration file along with setup files and requirements.
- “glue-scripts”: Contains the Python scripts which implement the actual ETL logic (processing/joining the sales and marketing datasets).
- “samples”: Provides sample CSV datasets used to trigger the ETL workflow.
- Additionally, the “build.py” script and Python Pynt tasks help package, deploy, and manage the stacks (create, update, delete).

##### Key Aspects from the README
- Detailed steps of the ETL workflow:
- Datasets for Sales and Marketing trigger wait states in the Step Functions state machine.
- AWS Lambda functions (termed “ETL Runners”) poll for activity tasks from the state machine.
- Once triggered, the Lambda functions invoke a corresponding ETL job (using AWS Glue or Amazon Athena).
- The Glue Runner Lambda also uses a DynamoDB table to track active jobs.
- CloudFormation templates are used to provision and configure AWS resources.
- The README provides example CLI commands and deployment instructions that confirm how the pieces are integrated.

- Clear labels for each major AWS component (Step Functions, Lambda (gluerunner/athenarunner/ons3objectcreated), Glue jobs, DynamoDB, S3 buckets, CloudFormation).
· The flow of triggers (e.g., S3 file uploads to Lambda).
· The initiation of activity tasks from Step Functions to the Lambda ETL Runners.
· Data flows between S3, AWS Glue, and DynamoDB.
· Control components (state machine, CloudFormation).
· Data processing components (ETL jobs in Glue/Athena).
· Data storage (S3 buckets, DynamoDB).
· Event sources/handlers (CloudWatch Events, S3 event-driven Lambda).

##### Final Assembly
- Start at the top with CloudFormation as the foundation, provisioning S3, Step Functions, Lambda functions, DynamoDB, and Glue resources.
- Next, detail the ETL orchestration:
- The Step Functions state machine acts as the command center.
- Parallel branches show different ETL jobs waiting for S3 uploads.
- ETL runners (Lambda functions) poll for tasks from Step Functions, start Glue jobs, and track state.
- Joined datasets and intermediary outputs saved to designated S3 prefixes.
- Error-handling and notifications (if any) via AWS CloudWatch log streams.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Provisioning Layer
    subgraph "Provisioning Layer"
        CF["AWS CloudFormation"]:::control
        BD["Build & Deploy (build.py)"]:::control
    end

    %% Event Sources
    subgraph "Event Sources"
        S3E["S3 Data Input (Samples)"]:::storage
        CW["CloudWatch Events"]:::event
    end

    %% Orchestration Layer
    subgraph "Orchestration"
        SF["Step Functions (MarketingAndSalesETLOrchestrator)"]:::control
    end

    %% AWS Lambda Functions
    subgraph "Lambda Functions"
        L_O3["Lambda OnS3ObjectCreated"]:::event
        L_GR["Lambda Glue Runner"]:::lambda
        L_AR["Lambda Athena Runner"]:::lambda
    end

    %% ETL Jobs Layer
    subgraph "ETL Jobs"
        Glue1["Glue Job: Process Sales Data"]:::etl
        Glue2["Glue Job: Process Marketing Data"]:::etl
        Glue3["Glue Job: Join Datasets"]:::etl
    end

    %% Data Storage Layer
    subgraph "Data Storage"
        S3["S3 Bucket (Input/Output)"]:::storage
        DDB["AWS DynamoDB"]:::database
    end

    %% Connections from Provisioning
    CF -->|"provisions"| SF
    CF -->|"provisions"| L_O3
    CF -->|"provisions"| L_GR
    CF -->|"provisions"| L_AR
    CF -->|"provisions"| S3
    CF -->|"provisions"| DDB
    CF -->|"provisions"| Glue1
    CF -->|"provisions"| Glue2
    CF -->|"provisions"| Glue3
    BD -->|"initiates"| CF

    %% Event Triggers and Control Flow
    S3E -->|"S3Event"| L_O3
    CW -->|"triggers"| SF
    L_O3 -->|"triggers"| SF

    SF -->|"dispatchTask"| L_GR
    SF -->|"dispatchTask"| L_AR

    L_GR -->|"triggers"| Glue1
    L_GR -->|"triggers"| Glue2
    L_GR -->|"updatesStatus"| DDB

    Glue1 -->|"feedsData"| Glue3
    Glue2 -->|"feedsData"| Glue3
    Glue3 -->|"writesOutput"| S3

    L_AR -->|"writesOutput"| S3

    %% Click Events
    click CF "https://github.com/aws-samples/aws-etl-orchestrator/tree/master/cloudformation"
    click SF "https://github.com/aws-samples/aws-etl-orchestrator/blob/master/cloudformation/step-functions-resources.yaml"
    click L_GR "https://github.com/aws-samples/aws-etl-orchestrator/tree/master/lambda/gluerunner"
    click L_AR "https://github.com/aws-samples/aws-etl-orchestrator/tree/master/lambda/athenarunner"
    click L_O3 "https://github.com/aws-samples/aws-etl-orchestrator/tree/master/lambda/ons3objectcreated"
    click Glue1 "https://github.com/aws-samples/aws-etl-orchestrator/blob/master/glue-scripts/process_sales_data.py"
    click Glue2 "https://github.com/aws-samples/aws-etl-orchestrator/blob/master/glue-scripts/process_marketing_data.py"
    click Glue3 "https://github.com/aws-samples/aws-etl-orchestrator/blob/master/glue-scripts/join_marketing_and_sales_data.py"
    click S3E "https://github.com/aws-samples/aws-etl-orchestrator/tree/master/samples"
    click BD "https://github.com/aws-samples/aws-etl-orchestrator/blob/master/build.py"

    %% Class Definitions for Colors
    classDef control fill:#FFEEAD,stroke:#D4A017,stroke-width:2px;
    classDef lambda fill:#AED6F1,stroke:#2471A3,stroke-width:2px;
    classDef etl fill:#A9DFBF,stroke:#1E8449,stroke-width:2px;
    classDef storage fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef database fill:#D2B4DE,stroke:#7D3C98,stroke-width:2px;
    classDef event fill:#F5CBA7,stroke:#AF601A,stroke-width:2px;
```
