# gabrielamazonas/airflow-pyspark-emr architecture

> Architecture diagram of the GitHub repository gabrielamazonas/airflow-pyspark-emr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gabrielamazonas/airflow-pyspark-emr
- Repository: https://github.com/gabrielamazonas/airflow-pyspark-emr

## Overview

##### Project Type and Purpose
- This is a data‐engineering project implementing an ETL pipeline (“songs_etl” and “terminate_emr_cluster” DAGs) using Apache Airflow to orchestrate PySpark jobs on AWS EMR, with raw and processed data in S3. The goal is to build a star‐schema OLAP data lake from JSON logs and song metadata.

##### Main Components
a. Airflow Control Plane
- Scheduler & Webserver: runs locally (Docker or virtualenv).
- DAGs directory (`airflow_home/dags`):
- songs_etl.py — builds EMR cluster, submits Spark job, watches for success.
- terminate_emr_cluster.py — tears down EMR cluster.
- Plugins (`airflow_home/plugins`): AWS utilities (EMR, IAM, S3) and Spark job stubs.
b. AWS Data Plane
- S3 Buckets:
- raw logs (`log_data/...`)
- raw song metadata (`song_data/...`)
- processed Parquet tables (star schema)
- EMR Cluster: Created by Airflow via boto3, runs PySpark on `airflow_home/spark_jobs/etl.py`, then terminated.
c. Spark Job
- PySpark logic (`etl.py`) reading from S3, transforming into fact and dimension tables, writing back to S3 in Parquet.

##### Relationships & Data Flow
1. Operator (or scheduler) triggers Airflow DAG.
2. songs_etl DAG uses AWS utils plugin to CREATE EMR cluster.
3. Airflow SUBMITS spark‐submit job on EMR cluster pointing to `etl.py`.
4. Spark reads raw JSON from S3, processes into star schema.
5. Spark writes Parquet tables back to S3.
6. DAG succeeds; terminate_emr_cluster DAG tears down EMR cluster.
7. Processed data in S3 is ready for downstream analytics (e.g., Athena, Redshift Spectrum).

##### Architectural Patterns & Principles
- Orchestration Pattern: Airflow DAGs decouple scheduling and job execution.
- Infrastructure as Code: plugins encapsulate AWS resource management (EMR, IAM, S3).
- Modular Design: Separate directories for DAGs, plugins (aws_utils, spark_jobs), and config.
- Scalability: EMR cluster sizing adjustable; scheduling frequency configurable.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Airflow Host
    subgraph "Airflow Host"
        Scheduler["Scheduler"]:::orchestration
        Webserver["Webserver"]:::orchestration

        subgraph "DAGs Directory (airflow_home/dags)"
            SongsETL["songs_etl DAG"]:::orchestration
            TerminateClusterDAG["terminate_emr_cluster DAG"]:::orchestration
        end

        subgraph "Plugins Directory (airflow_home/plugins/aws_utils)"
            EMROps["EMR Operations"]:::orchestration
            IAMOps["IAM Operations"]:::orchestration
            S3Ops["S3 Operations"]:::orchestration
        end

        Config["dl.cfg"]:::config
        Helper["config.sh"]:::config
        DockerfileNode["Dockerfile"]:::config
        StartScript["start.sh"]:::config
        ReqTxt["requirements.txt"]:::config
    end

    subgraph "Spark Jobs Directory (airflow_home/spark_jobs)"
        ETLJob["etl.py"]:::compute
    end

    subgraph "AWS Cloud"
        S3Raw["Raw JSON S3 Bucket"]:::storage
        S3Processed["Processed Parquet S3 Bucket"]:::storage
        EMRCluster["EMR Cluster"]:::compute
        IAMRole["IAM Role"]:::orchestration
    end

    %% Connections
    Scheduler -->|schedules| SongsETL
    SongsETL -->|uses| EMROps
    EMROps -->|CreateCluster| EMRCluster
    SongsETL -->|spark-submit| EMRCluster
    EMRCluster -->|reads JSON| S3Raw
    EMRCluster -->|runs etl.py| ETLJob
    ETLJob -->|writes Parquet| S3Processed
    ETLJob -->|reads config| Config
    SongsETL -->|on success triggers| TerminateClusterDAG
    TerminateClusterDAG -->|uses| EMROps
    EMROps -->|TerminateCluster| EMRCluster

    SongsETL -->|assumes role| IAMOps
    IAMOps -->|manages| IAMRole
    SongsETL -->|uses| S3Ops
    S3Ops -->|accesses| S3Raw
    S3Ops -->|accesses| S3Processed

    DockerfileNode -->|builds| Scheduler
    DockerfileNode -->|builds| Webserver
    StartScript -->|starts| Scheduler
    StartScript -->|starts| Webserver
    Config -->|provides creds| SongsETL
    Config -->|provides creds| TerminateClusterDAG
    Config -->|provides creds| ETLJob
    ReqTxt -->|installs deps| Scheduler
    ReqTxt -->|installs deps| Webserver
    ReqTxt -->|installs deps| SongsETL
    ReqTxt -->|installs deps| EMROps

    %% Click events
    click SongsETL "https://github.com/gabrielamazonas/airflow-pyspark-emr/blob/master/airflow_home/dags/songs_etl.py"
    click TerminateClusterDAG "https://github.com/gabrielamazonas/airflow-pyspark-emr/blob/master/airflow_home/dags/terminate_emr_cluster.py"
    click EMROps "https://github.com/gabrielamazonas/airflow-pyspark-emr/blob/master/airflow_home/plugins/aws_utils/emr_operations.py"
    click IAMOps "https://github.com/gabrielamazonas/airflow-pyspark-emr/blob/master/airflow_home/plugins/aws_utils/iam_operations.py"
    click S3Ops "https://github.com/gabrielamazonas/airflow-pyspark-emr/blob/master/airflow_home/plugins/aws_utils/s3_operations.py"
    click ETLJob "https://github.com/gabrielamazonas/airflow-pyspark-emr/blob/master/airflow_home/spark_jobs/etl.py"
    click Config "https://github.com/gabrielamazonas/airflow-pyspark-emr/blob/master/airflow_home/dl.cfg"
    click Helper "https://github.com/gabrielamazonas/airflow-pyspark-emr/blob/master/config.sh"
    click DockerfileNode "https://github.com/gabrielamazonas/airflow-pyspark-emr/tree/master/Dockerfile"
    click StartScript "https://github.com/gabrielamazonas/airflow-pyspark-emr/blob/master/start.sh"
    click ReqTxt "https://github.com/gabrielamazonas/airflow-pyspark-emr/blob/master/requirements.txt"

    %% Styles
    classDef orchestration fill:#AED6F1,stroke:#333,stroke-width:1px
    classDef compute fill:#F5B041,stroke:#333,stroke-width:1px
    classDef storage fill:#ABEBC6,stroke:#333,stroke-width:1px
    classDef config fill:#D7DBDD,stroke:#333,stroke-width:1px
```
