How longbuivan/airflow-spark-data-pipeline works
Main Components
a) Data Sources Layer:
• Video views data source (CSV format)
• Videos metadata source (JSON format)
b) Processing Infrastructure:
• Apache Airflow Orchestration
• Airflow Webserver (Port 8080)
• DAG components (from dags/spark-view-percentage.py)
• Spark Cluster
• Spark Master (Port 7077)
• Two Spark Workers
• Spark Submit jobs
• Storage Layer
• PostgreSQL Databases
• Airflow metadata DB
• Test DB
c) Development Environment:
• Jupyter Notebook environment with PySpark
Component Relationships
• Control flow from Airflow to Spark Master
• Data flow between Spark Master and Workers
• Database connections from both Airflow and Spark
• Development environment connections to Spark cluster
Specific Implementation Details
• Containers (rectangles with docker logo)
• Processing units (hexagons for Spark components)
• Orchestration components (rounded rectangles for Airflow)
Technical Details
• Port mappings (8080, 7077, 5432, etc.)
• Volume mounts for data persistence
• Network connections between components
Processing Flow Indicators
Data processing pipeline:
Infrastructure Details
A separate section showing:
• Container orchestration with Docker Compose
• Resource allocations (executor cores, memory)
• Network configuration
Key Considerations:
Technical Specifications
• Spark configuration (--num-executors 2, --executor-cores 1, --executor-memory 1024M)
• Database credentials (but marked as configurable)
• Container versions (Spark 3.1.2, Airflow 1.10.9, Postgres 9.6)