# akashdv25/pyspark-eda-flights-data architecture

> Architecture diagram of the GitHub repository akashdv25/pyspark-eda-flights-data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/akashdv25/pyspark-eda-flights-data
- Repository: https://github.com/akashdv25/pyspark-eda-flights-data

## Overview

##### Project Type and Purpose
- This is a batch-oriented ETL/data processing pipeline built with PySpark.
- It ingests raw flight, airline, and airport CSV data from AWS S3, cleans and enriches it, creates a down-sampled subset, writes outputs back to S3, and supports interactive analysis via a Jupyter notebook.

##### File Structure
- Top-level directories:
- src/ – contains five modules, each implementing one pipeline stage
- notebook/ – houses the Jupyter notebook for exploration
- docs/ – requirements and internal documentation
- Key scripts in src/:
- data_ingestion.py (SparkSession setup, reads from S3)
- data_cleaning.py (null handling, deduplication)
- data_enrichment.py (broadcast joins, derived metrics)
- subset_flights_df.py (sampling)
- upload_csv_to_s3.py (writes Parquet/CSV to S3)

##### README for Additional Insights
- The README outlines a linear data-processing pipeline: ingestion → cleaning → enrichment → sampling → export → analysis.
- It calls out AWS S3 integration, Spark performance optimizations (broadcast joins, caching), and native vs UDF performance comparisons.

##### Detailed but Focused
- External boundaries (local orchestrator, AWS S3, Spark cluster, notebook)
- Call out configuration files (example.env, requirements.txt) as side notes or a small “config” component

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "AWS S3 Storage"
        RawData["Raw CSV Data"]:::s3
        ProcessedData["Processed Outputs"]:::s3
    end

    SparkCluster[(Spark Cluster)]:::spark

    subgraph "ETL Pipeline"
        direction TB
        Ingestion["data_ingestion.py: reads from S3"]:::proc
        Cleaning["data_cleaning.py: clean_nulls(), remove_duplicates()"]:::proc
        Enrichment["data_enrichment.py: broadcast joins, derived metrics"]:::proc
        Sampling["subset_flights_df.py: sampling"]:::proc
        Export["upload_csv_to_s3.py: writes Parquet/CSV to S3"]:::proc
    end

    subgraph "Supporting Components"
        direction TB
        Orchestrator["Local Orchestration\n(virtualenv, AWS CLI)"]:::config
        ExampleEnv["example.env"]:::config
        Requirements["requirements.txt"]:::config
        Docs["docs/"]:::config
        Img["img/"]:::config
        README["README.md"]:::config
    end

    Notebook["flights_data_analysis.ipynb"]:::note

    RawData --> Ingestion
    Ingestion --> Cleaning
    Cleaning --> Enrichment
    Enrichment --> Sampling
    Sampling --> Export
    Export --> ProcessedData

    SparkCluster --- Ingestion
    SparkCluster --- Cleaning
    SparkCluster --- Enrichment
    SparkCluster --- Sampling
    SparkCluster --- Export

    Orchestrator --> Ingestion

    ProcessedData --> Notebook
    Notebook --- SparkCluster

    click Ingestion "https://github.com/akashdv25/pyspark-eda-flights-data/blob/main/src/data_ingestion.py"
    click Cleaning "https://github.com/akashdv25/pyspark-eda-flights-data/blob/main/src/data_cleaning.py"
    click Enrichment "https://github.com/akashdv25/pyspark-eda-flights-data/blob/main/src/data_enrichment.py"
    click Sampling "https://github.com/akashdv25/pyspark-eda-flights-data/blob/main/src/subset_flights_df.py"
    click Export "https://github.com/akashdv25/pyspark-eda-flights-data/blob/main/src/upload_csv_to_s3.py"
    click Notebook "https://github.com/akashdv25/pyspark-eda-flights-data/blob/main/notebook/flights_data_analysis.ipynb"
    click ExampleEnv "https://github.com/akashdv25/pyspark-eda-flights-data/blob/main/example.env"
    click Requirements "https://github.com/akashdv25/pyspark-eda-flights-data/blob/main/requirements.txt"
    click Docs "https://github.com/akashdv25/pyspark-eda-flights-data/tree/main/docs/"
    click Img "https://github.com/akashdv25/pyspark-eda-flights-data/tree/main/img/"
    click README "https://github.com/akashdv25/pyspark-eda-flights-data/blob/main/README.md"

    classDef s3 fill:#FF9933,stroke:#333,stroke-width:1px
    classDef proc fill:#3399FF,stroke:#333,stroke-width:1px
    classDef spark fill:#33CC33,stroke:#333,stroke-width:1px
    classDef note fill:#CC99FF,stroke:#333,stroke-width:1px
    classDef config fill:#CCCCCC,stroke:#333,stroke-width:1px
```
