# longbuivan/airflow-spark-data-pipeline architecture

> Architecture diagram of the GitHub repository longbuivan/airflow-spark-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/longbuivan/airflow-spark-data-pipeline
- Repository: https://github.com/longbuivan/airflow-spark-data-pipeline

## Overview

##### Main Components

a) Data Sources Layer:
- Video views data source (CSV format)
- Videos metadata source (JSON format)

b) Processing Infrastructure:
- Apache Airflow Orchestration
- Airflow Webserver (Port 8080)
- DAG components (from dags/spark-view-percentage.py)

- Spark Cluster
- Spark Master (Port 7077)
- Two Spark Workers
- Spark Submit jobs

- Storage Layer
- PostgreSQL Databases
- Airflow metadata DB
- Test DB

c) Development Environment:
- Jupyter Notebook environment with PySpark

##### Component Relationships

- Control flow from Airflow to Spark Master
- Data flow between Spark Master and Workers
- Database connections from both Airflow and Spark
- Development environment connections to Spark cluster

##### Specific Implementation Details

- Containers (rectangles with docker logo)
- Processing units (hexagons for Spark components)
- Orchestration components (rounded rectangles for Airflow)

##### Technical Details
- Port mappings (8080, 7077, 5432, etc.)
- Volume mounts for data persistence
- Network connections between components

##### Processing Flow Indicators

Data processing pipeline:

##### Infrastructure Details

A separate section showing:
- Container orchestration with Docker Compose
- Resource allocations (executor cores, memory)
- Network configuration

Key Considerations:

##### Technical Specifications
- Spark configuration (--num-executors 2, --executor-cores 1, --executor-memory 1024M)
- Database credentials (but marked as configurable)
- Container versions (Spark 3.1.2, Airflow 1.10.9, Postgres 9.6)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and Classes
    classDef source fill:#e1f5fe,stroke:#01579b
    classDef airflow fill:#e8f5e9,stroke:#2e7d32
    classDef spark fill:#fff3e0,stroke:#e65100
    classDef storage fill:#f3e5f5,stroke:#4a148c
    classDef ui fill:#fce4ec,stroke:#880e4f
    classDef dev fill:#f5f5f5,stroke:#212121

    %% Client Layer
    subgraph ClientLayer
        AirflowUI["Airflow UI\n(Port: 8080)"]:::ui
        JupyterUI["Jupyter Notebook\nDevelopment Interface"]:::ui
    end

    %% Data Sources
    subgraph DataSources
        CSV["Video Views CSV"]:::source
        JSON["Videos JSON"]:::source
    end

    %% Orchestration Layer
    subgraph OrchestrationLayer
        AirflowScheduler["Airflow Scheduler"]:::airflow
        AirflowWorker["Airflow Worker"]:::airflow
        ViewPercentageDAG["View Percentage DAG"]:::airflow
    end

    %% Processing Layer
    subgraph SparkCluster
        SparkMaster["Spark Master\n(Port: 7077)"]:::spark
        Worker1["Spark Worker 1"]:::spark
        Worker2["Spark Worker 2"]:::spark
        CalcData["Calculate Data"]:::spark
        LoadData["Load Data"]:::spark
    end

    %% Storage Layer
    subgraph StorageLayer
        PostgreSQL["PostgreSQL\n(Port: 5432)"]:::storage
        AirflowDB["Airflow DB"]:::storage
        AppDB["Application DB"]:::storage
    end

    %% Development Tools
    subgraph DevTools
        TestWorker["Spark Tests"]:::dev
        LoadTest["Load Tests"]:::dev
    end

    %% Relationships
    CSV --> LoadData
    JSON --> LoadData
    LoadData --> Worker1
    LoadData --> Worker2
    Worker1 --> CalcData
    Worker2 --> CalcData
    CalcData --> PostgreSQL
    AirflowScheduler --> ViewPercentageDAG
    ViewPercentageDAG --> SparkMaster
    SparkMaster --> Worker1
    SparkMaster --> Worker2
    PostgreSQL --> AirflowDB
    PostgreSQL --> AppDB
    JupyterUI --> SparkMaster
    AirflowUI --> AirflowScheduler

    %% Click Events
    click CSV "https://github.com/longbuivan/airflow-spark-data-pipeline/blob/main/spark/resources/data/video_views_mini.csv"
    click JSON "https://github.com/longbuivan/airflow-spark-data-pipeline/blob/main/spark/resources/data/videos.json"
    click ViewPercentageDAG "https://github.com/longbuivan/airflow-spark-data-pipeline/blob/main/dags/spark-view-percentage.py"
    click CalcData "https://github.com/longbuivan/airflow-spark-data-pipeline/blob/main/spark/app/calculate-data.py"
    click LoadData "https://github.com/longbuivan/airflow-spark-data-pipeline/blob/main/spark/app/load-data.py"
    click JupyterUI "https://github.com/longbuivan/airflow-spark-data-pipeline/blob/main/notebooks/data_analysis.ipynb"
    click PostgreSQL "https://github.com/longbuivan/airflow-spark-data-pipeline/blob/main/spark/resources/jars/postgresql-9.4.1207.jar"
    click TestWorker "https://github.com/longbuivan/airflow-spark-data-pipeline/blob/main/spark/app/test_worker.py"
    click LoadTest "https://github.com/longbuivan/airflow-spark-data-pipeline/blob/main/spark/app/production_load_test.py"
```
