How shaalanmarwan/airline-delay-and-cancellation works
Data Sources Layer
• Kaggle dataset source (2009-2018 airline data)
• Supplementary data sources (airlines_lookup.csv and airports_lookup.csv)
Storage Layer
• Local Storage (CSV files)
• Google Cloud Storage (Parquet files)
Processing Layer (show as separate connected components)
a. ETL Pipeline:
• Python processing scripts (etl_to_gcs.py and etl_gcs_to_bq.py)
• Flights delay transformer module
b. DBT Transformation Layer:
• Staging models (stg_flightdata)
• Core models (airlines_zone, airports_zone, fact_flights)
• Clear data flow between models
Data Warehouse Layer
• BigQuery with partitioned and clustered tables
• Partition by FL_DATE
• Clustering by OP_CARRIER, ORIGIN, DEST
Visualization Layer
• Google Data Studio dashboards
• Two main dashboard pages (Flights Delay and Flights Cancellations)
Design Guidelines:
• Data flow from source to final visualization
• ETL process steps
• DBT model dependencies
Component Grouping
• Separate infrastructure components (Terraform) from data flow
• Clear boundaries between different processing stages
Infrastructure Elements
• Google Cloud Platform components
• Terraform-managed resources
• Cloud services boundaries
Additional Elements
• Version control (Git) integration points
• Testing components and data quality checks
Specific Implementation Details:
• [Data Sources] → [Storage] → [Processing] → [Data Warehouse] → [Visualization]
Specific technical details
• File formats (CSV → Parquet → BigQuery tables)
• DBT model hierarchy
• Partitioning and clustering specifications
Infrastructure Components
• Google Cloud Platform services
• Network boundaries and security zones
Data Quality and Testing
• DBT tests location
• Data validation points
• Quality check processes