How priyanshubiswas-tech/aws-etl-pipeline-on-cloud-using-glue-athena-lambda-and-redshift works
Project Type and Purpose
• This is a serverless, event-driven ETL pipeline hosted on AWS.
• Its purpose is to ingest raw CSV data from S3, infer schema via Glue Crawler, transform data with a Glue PySpark job, validate with Athena, and load into Redshift—all orchestrated via Lambda and managed via CloudFormation/IAM.
File Structure
• Top-level directories map directly to architectural components:
• cloudformation/ → IaC definitions for S3, Glue, Lambda, Redshift
• glue_jobs/ → PySpark ETL script + crawler configuration
• lambda/ → event-trigger Lambda function
• sql/ → DDL & validation queries for Athena/Redshift
• iam_roles/ → IAM policy JSON granting permissions
• data_samples/ → example input data (to orient S3 source)
• There are no “monolith” code pools—everything is decoupled AWS service resources.
README for Additional Insights
• The README already sketches the sequence: S3 → Glue Crawler → Glue Job → Athena → Lambda → Redshift
• It calls out key services: S3 Bucket, AWS Glue Crawler & Job, Athena, Lambda Trigger, Redshift cluster, IAM roles, CloudFormation stack.
Main Components
a. S3 Bucket (raw CSV data)
b. AWS Glue Crawler (populates Data Catalog)
c. AWS Glue ETL Job (PySpark transform & write to Redshift)
d. AWS Lambda (listens to S3 events, starts Glue job)
e. AWS Athena (validation queries against transformed data or raw catalog)
f. Amazon Redshift (final data store)
g. IAM Roles & Policies (security boundary around Glue, Lambda, Redshift, S3)
h. CloudFormation Stack (infrastructure provisioning)
Relationships and Interactions
S3 → (S3 event) → Lambda → starts → Glue Crawler & Glue Job → transforms & writes → Redshift
Glue Crawler also writes metadata into Glue Data Catalog (used by Athena)
Athena queries Data Catalog (raw/transformed tables)
• Directionality:
Architectural Patterns and Design Principles
• Serverless & fully managed: no EC2, auto-scaling by AWS services
• Event-driven orchestration: S3 events → Lambda → Glue
• Modular & decoupled: each component stands alone, permissions scoped via IAM policies
• Infrastructure as Code: all resources declared in cloudformation/etl_pipeline_stack.yaml
Relevant Technologies, Frameworks, and Libraries
• Python/PySpark for transformations (glue_jobs/job_transform.py)
• AWS Glue Data Catalog (crawler metadata store)
• AWS SDK (boto3) in Lambda (lambda/lambda_trigger.py)
• SQL DDL & validation queries (sql/redshift_table_ddl.sql, sql/validation_queries.sql)
• CloudFormation (cloudformation/etl_pipeline_stack.yaml)
• IAM policies (iam_roles/glue_lambda_redshift_policies.json)