# shaalanmarwan/airline-delay-and-cancellation architecture

> Architecture diagram of the GitHub repository shaalanmarwan/airline-delay-and-cancellation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shaalanmarwan/airline-delay-and-cancellation
- Repository: https://github.com/shaalanmarwan/airline-delay-and-cancellation

## Overview

##### Data Sources Layer
- Kaggle dataset source (2009-2018 airline data)
- Supplementary data sources (airlines_lookup.csv and airports_lookup.csv)

##### Storage Layer
- Local Storage (CSV files)
- Google Cloud Storage (Parquet files)

##### Processing Layer (show as separate connected components)
a. ETL Pipeline:
- Python processing scripts (etl_to_gcs.py and etl_gcs_to_bq.py)
- Flights delay transformer module
b. DBT Transformation Layer:
- Staging models (stg_flightdata)
- Core models (airlines_zone, airports_zone, fact_flights)
- Clear data flow between models

##### Data Warehouse Layer
- BigQuery with partitioned and clustered tables
- Partition by FL_DATE
- Clustering by OP_CARRIER, ORIGIN, DEST

##### Visualization Layer
- Google Data Studio dashboards
- Two main dashboard pages (Flights Delay and Flights Cancellations)

Design Guidelines:

- Data flow from source to final visualization
- ETL process steps
- DBT model dependencies

##### Component Grouping
- Separate infrastructure components (Terraform) from data flow
- Clear boundaries between different processing stages

##### Infrastructure Elements
- Google Cloud Platform components
- Terraform-managed resources
- Cloud services boundaries

##### Additional Elements
- Version control (Git) integration points
- Testing components and data quality checks

Specific Implementation Details:

- [Data Sources] → [Storage] → [Processing] → [Data Warehouse] → [Visualization]

##### Specific technical details
- File formats (CSV → Parquet → BigQuery tables)
- DBT model hierarchy
- Partitioning and clustering specifications

##### Infrastructure Components
- Google Cloud Platform services
- Network boundaries and security zones

##### Data Quality and Testing
- DBT tests location
- Data validation points
- Quality check processes

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and Classes
    classDef source fill:#90EE90,stroke:#333,stroke-width:2px
    classDef storage fill:#FFFF99,stroke:#333,stroke-width:2px
    classDef processing fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef dbt fill:#DDA0DD,stroke:#333,stroke-width:2px
    classDef viz fill:#FFA500,stroke:#333,stroke-width:2px
    classDef infra fill:#FFB6C1,stroke:#333,stroke-width:2px

    %% Data Sources Layer
    subgraph DataSources["Data Sources Layer"]
        KaggleData["Kaggle Flight Data<br/>(2009-2018)"]:::source
        AirlinesLookup["Airlines Lookup CSV"]:::source
        AirportsLookup["Airports Lookup CSV"]:::source
    end

    %% Storage Layer
    subgraph Storage["Storage Layer"]
        LocalCSV["Local CSV Files"]:::storage
        GCS["Google Cloud Storage<br/>(Parquet)"]:::storage
    end

    %% Processing Layer
    subgraph Processing["Processing Layer"]
        ETL_GCS["ETL to GCS"]:::processing
        ETL_BQ["ETL GCS to BigQuery"]:::processing
        Transformer["Flights Delay<br/>Transformer"]:::processing
        TestModule["Data Quality Tests"]:::processing
    end

    %% DBT Layer
    subgraph DBT["DBT Transformation Layer"]
        StagingModel["Staging Model<br/>stg_flightdata"]:::dbt
        AirlinesZone["Airlines Zone"]:::dbt
        AirportsZone["Airports Zone"]:::dbt
        FactFlights["Fact Flights"]:::dbt
    end

    %% Data Warehouse Layer
    subgraph Warehouse["Data Warehouse"]
        BQ["BigQuery<br/>(Partitioned by FL_DATE<br/>Clustered by OP_CARRIER,<br/>ORIGIN, DEST)"]:::storage
    end

    %% Visualization Layer
    subgraph Visualization["Visualization Layer"]
        GDS["Google Data Studio<br/>Dashboards"]:::viz
        DelayDash["Flights Delay<br/>Dashboard"]:::viz
        CancelDash["Flights Cancellation<br/>Dashboard"]:::viz
    end

    %% Infrastructure
    subgraph Infrastructure["Infrastructure"]
        Terraform["Terraform Config"]:::infra
        DBTConfig["DBT Config"]:::infra
    end

    %% Relationships
    KaggleData --> LocalCSV
    AirlinesLookup --> LocalCSV
    AirportsLookup --> LocalCSV
    LocalCSV --> ETL_GCS
    ETL_GCS --> GCS
    GCS --> ETL_BQ
    ETL_BQ --> Transformer
    Transformer --> BQ
    TestModule --> Transformer
    BQ --> StagingModel
    StagingModel --> AirlinesZone
    StagingModel --> AirportsZone
    AirlinesZone --> FactFlights
    AirportsZone --> FactFlights
    FactFlights --> GDS
    GDS --> DelayDash
    GDS --> CancelDash
    Terraform -.-> GCS
    Terraform -.-> BQ
    DBTConfig -.-> DBT

    %% Click Events
    click AirlinesLookup "https://github.com/ShaalanMarwan/airline-delay-and-cancellation/blob/main/dbt/seeds/airlines_lookup.csv"
    click AirportsLookup "https://github.com/ShaalanMarwan/airline-delay-and-cancellation/blob/main/dbt/seeds/airports_lookup.csv"
    click ETL_GCS "https://github.com/ShaalanMarwan/airline-delay-and-cancellation/blob/main/dbt/processing/flow/etl_to_gcs.py"
    click ETL_BQ "https://github.com/ShaalanMarwan/airline-delay-and-cancellation/blob/main/dbt/processing/flow/etl_gcs_to_bq.py"
    click Transformer "https://github.com/ShaalanMarwan/airline-delay-and-cancellation/blob/main/dbt/processing/helper/flights_delay_transformer_module.py"
    click StagingModel "https://github.com/ShaalanMarwan/airline-delay-and-cancellation/blob/main/dbt/models/staging/stg_flightdata.sql"
    click AirlinesZone "https://github.com/ShaalanMarwan/airline-delay-and-cancellation/blob/main/dbt/models/core/airlines_zone.sql"
    click AirportsZone "https://github.com/ShaalanMarwan/airline-delay-and-cancellation/blob/main/dbt/models/core/airports_zone.sql"
    click FactFlights "https://github.com/ShaalanMarwan/airline-delay-and-cancellation/blob/main/dbt/models/core/fact_flights.sql"
    click TestModule "https://github.com/ShaalanMarwan/airline-delay-and-cancellation/blob/main/dbt/processing/helper/test_flights_delay_transformer.py"
    click Terraform "https://github.com/ShaalanMarwan/airline-delay-and-cancellation/tree/main/terraform"
    click DBTConfig "https://github.com/ShaalanMarwan/airline-delay-and-cancellation/blob/main/dbt/dbt_project.yml"
```
