How gabrielamazonas/airflow-pyspark-emr works
Project Type and Purpose
• This is a data‐engineering project implementing an ETL pipeline (“songs_etl” and “terminate_emr_cluster” DAGs) using Apache Airflow to orchestrate PySpark jobs on AWS EMR, with raw and processed data in S3. The goal is to build a star‐schema OLAP data lake from JSON logs and song metadata.
Main Components
a. Airflow Control Plane
• Scheduler & Webserver: runs locally (Docker or virtualenv).
• DAGs directory (airflow_home/dags):
• songs_etl.py — builds EMR cluster, submits Spark job, watches for success.
• terminate_emr_cluster.py — tears down EMR cluster.
• Plugins (airflow_home/plugins): AWS utilities (EMR, IAM, S3) and Spark job stubs.
b. AWS Data Plane
• S3 Buckets:
• raw logs (log_data/...)
• raw song metadata (song_data/...)
• processed Parquet tables (star schema)
• EMR Cluster: Created by Airflow via boto3, runs PySpark on airflow_home/spark_jobs/etl.py, then terminated.
c. Spark Job
• PySpark logic (etl.py) reading from S3, transforming into fact and dimension tables, writing back to S3 in Parquet.
Relationships & Data Flow
1. Operator (or scheduler) triggers Airflow DAG.
2. songs_etl DAG uses AWS utils plugin to CREATE EMR cluster.
3. Airflow SUBMITS spark‐submit job on EMR cluster pointing to etl.py.
4. Spark reads raw JSON from S3, processes into star schema.
5. Spark writes Parquet tables back to S3.
6. DAG succeeds; terminate_emr_cluster DAG tears down EMR cluster.
7. Processed data in S3 is ready for downstream analytics (e.g., Athena, Redshift Spectrum).
Architectural Patterns & Principles
• Orchestration Pattern: Airflow DAGs decouple scheduling and job execution.
• Infrastructure as Code: plugins encapsulate AWS resource management (EMR, IAM, S3).
• Modular Design: Separate directories for DAGs, plugins (aws_utils, spark_jobs), and config.
• Scalability: EMR cluster sizing adjustable; scheduling frequency configurable.