# priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift architecture

> Architecture diagram of the GitHub repository priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift
- Repository: https://github.com/priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift

## Overview

##### Project Type and Purpose
- This is a serverless, event-driven ETL pipeline hosted on AWS.
- Its purpose is to ingest raw CSV data from S3, infer schema via Glue Crawler, transform data with a Glue PySpark job, validate with Athena, and load into Redshift—all orchestrated via Lambda and managed via CloudFormation/IAM.

##### File Structure
- Top-level directories map directly to architectural components:
- cloudformation/ → IaC definitions for S3, Glue, Lambda, Redshift
- glue_jobs/ → PySpark ETL script + crawler configuration
- lambda/ → event-trigger Lambda function
- sql/ → DDL & validation queries for Athena/Redshift
- iam_roles/ → IAM policy JSON granting permissions
- data_samples/ → example input data (to orient S3 source)
- There are no “monolith” code pools—everything is decoupled AWS service resources.

##### README for Additional Insights
- The README already sketches the sequence: S3 → Glue Crawler → Glue Job → Athena → Lambda → Redshift
- It calls out key services: S3 Bucket, AWS Glue Crawler & Job, Athena, Lambda Trigger, Redshift cluster, IAM roles, CloudFormation stack.

##### Main Components
a. S3 Bucket (raw CSV data)
b. AWS Glue Crawler (populates Data Catalog)
c. AWS Glue ETL Job (PySpark transform & write to Redshift)
d. AWS Lambda (listens to S3 events, starts Glue job)
e. AWS Athena (validation queries against transformed data or raw catalog)
f. Amazon Redshift (final data store)
g. IAM Roles & Policies (security boundary around Glue, Lambda, Redshift, S3)
h. CloudFormation Stack (infrastructure provisioning)

##### Relationships and Interactions
S3 → (S3 event) → Lambda → starts → Glue Crawler & Glue Job → transforms & writes → Redshift
Glue Crawler also writes metadata into Glue Data Catalog (used by Athena)
Athena queries Data Catalog (raw/transformed tables)
- Directionality:

##### Architectural Patterns and Design Principles
- Serverless & fully managed: no EC2, auto-scaling by AWS services
- Event-driven orchestration: S3 events → Lambda → Glue
- Modular & decoupled: each component stands alone, permissions scoped via IAM policies
- Infrastructure as Code: all resources declared in cloudformation/etl_pipeline_stack.yaml

##### Relevant Technologies, Frameworks, and Libraries
- Python/PySpark for transformations (glue_jobs/job_transform.py)
- AWS Glue Data Catalog (crawler metadata store)
- AWS SDK (boto3) in Lambda (lambda/lambda_trigger.py)
- SQL DDL & validation queries (sql/redshift_table_ddl.sql, sql/validation_queries.sql)
- CloudFormation (cloudformation/etl_pipeline_stack.yaml)
- IAM policies (iam_roles/glue_lambda_redshift_policies.json)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "CloudFormation Stack"
        S3["S3 Bucket\n(raw CSV data)"]:::storage
        Lambda["AWS Lambda\n(event trigger)"]:::compute
        GlueCrawler["AWS Glue Crawler"]:::compute
        GlueJob["Glue ETL Job\n(job_transform.py)"]:::compute
        DataCatalog["Glue Data Catalog"]:::storage
        Athena["AWS Athena"]:::query
        Redshift["Amazon Redshift\n(final store)"]:::database
        IAM["IAM Roles & Policies"]:::boundary
    end

    Sample["Sample Input Data\n(sample_input_data.csv)"]:::storage

    Sample -->|"uploads"| S3
    S3 -->|"S3:ObjectCreated"| Lambda
    Lambda -.->|"Start Crawler"| GlueCrawler
    Lambda -.->|"Start ETL Job"| GlueJob
    GlueCrawler -->|"Writes metadata"| DataCatalog
    DataCatalog -->|"Catalog access"| Athena
    GlueJob -->|"JDBC write to Redshift"| Redshift
    Athena -->|"Validation queries"| Redshift

    IAM -.->|"Permissions"| S3
    IAM -.->|"Permissions"| Lambda
    IAM -.->|"Permissions"| GlueCrawler
    IAM -.->|"Permissions"| GlueJob
    IAM -.->|"Permissions"| Redshift
    IAM -.->|"Permissions"| Athena

    click S3 "https://github.com/priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift/blob/main/cloudformation/etl_pipeline_stack.yaml"
    click Lambda "https://github.com/priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift/blob/main/lambda/lambda_trigger.py"
    click GlueCrawler "https://github.com/priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift/blob/main/cloudformation/etl_pipeline_stack.yaml"
    click GlueCrawler "https://github.com/priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift/blob/main/glue_jobs/job_crawler_config.json"
    click GlueJob "https://github.com/priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift/blob/main/glue_jobs/job_transform.py"
    click Athena "https://github.com/priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift/blob/main/sql/validation_queries.sql"
    click Redshift "https://github.com/priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift/blob/main/sql/redshift_table_ddl.sql"
    click IAM "https://github.com/priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift/blob/main/iam_roles/glue_lambda_redshift_policies.json"
    click Sample "https://github.com/priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift/blob/main/data_samples/sample_input_data.csv"

    classDef compute fill:#BBDEFB,stroke:#000
    classDef storage fill:#FFF9C4,stroke:#000
    classDef database fill:#FFCDD2,stroke:#000
    classDef query fill:#C8E6C9,stroke:#000
    classDef boundary fill:#E0E0E0,stroke:#888,stroke-dasharray: 5 5
    class IAM boundary
```
