# hudiyaresa/airflow-dbt-flightdata architecture

> Architecture diagram of the GitHub repository hudiyaresa/airflow-dbt-flightdata, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hudiyaresa/airflow-dbt-flightdata
- Repository: https://github.com/hudiyaresa/airflow-dbt-flightdata

## Overview

##### Project Type & Purpose
- This is an orchestrated ELT (Extract, Load, Transform) pipeline built on Apache Airflow.
- It extracts flight‐booking data from a PostgreSQL source, lands CSVs in MinIO, loads into a staging Postgres schema, then transforms into dimensional/fact tables via DBT.
- It runs inside Docker Compose, uses CeleryExecutor for scaling, supports Slack alerting, Jinja templating, XComs and incremental loads.

##### File Structure & Key Artifacts
- Top-level:
- Dockerfile, docker-compose.yml, start.sh
- / (Airflow CLI variables & connections)
- dags/ (two DAGs: flights_staging_pipeline, flights_warehouse_pipeline)
- helper/ (MinIO & Postgres connectors, Slack callback)
- data/ (init SQL for source & warehouse)
- dbt‐requirements.txt, requirements.txt
- flights_staging_pipeline:
- run.py (DAG definition & scheduling)
- tasks/components: extract.py, load.py, transform.py (modular operators)
- flights_warehouse_pipeline:
- run.py triggers DBT (via TriggerDagRunOperator)
- pacflight_dbt/ (dbt_project.yml, snapshots, seeds, logs)
- Config:
- .env (Airflow and service credentials)
- /connections.yaml, include/variables.json loaded via start.sh

##### Extract Architectural Insights from README
- Orchestrator: Airflow (Scheduler, Webserver, Workers via CeleryExecutor)
- Source DB: PostgreSQL (bookings schema)
- Data Lake: MinIO (“extracted-data” bucket)
- Data Warehouse:
- Staging Schema: Postgres via Upsert (pangres)
- Warehouse Schema: DBT models (dimensional/fact tables)
- Notification: Slack Webhook (failure alerts)
- Deployment: Docker Compose

##### Main Components & Relationships
a. Components:
- Docker Compose environment
- Airflow Scheduler, Webserver, Celery Workers
- Redis (Broker), Postgres (Metadata DB)
- Source Postgres DB
- MinIO Object Store
- Staging Postgres DB
- DBT engine (inside Airflow task)
- Warehouse Postgres DB
- Slack endpoint
b. Interactions:
1. Scheduler triggers staging DAG
2. Extract task → reads from Source DB → writes CSV to MinIO
3. XCom passes file metadata → Load task pulls from MinIO → Upserts into Staging DB
4. TriggerDagRunOperator → kicks off warehouse DAG
5. Warehouse DAG runs DBT models against Staging → writes to Warehouse DB
6. On failure callbacks → helper.slack_notifier → Slack Webhook
c. Architectural Patterns:
- Modular task components (extract/load/transform)
- Dynamic task creation via Jinja templates
- XCom for lightweight metadata passing
- Upsert pattern for idempotent loads (pangres)
- TriggerDagRun for pipeline chaining
- DBT for transformations (snapshot/seed/model)
- CeleryExecutor for distributed task execution
d. Technologies & Frameworks:
- Python + Airflow
- Docker Compose, Celery, Redis
- PostgreSQL, MinIO
- DBT, Pangres
- Slack Webhooks

##### Essential Elements
- On environment/config loading (start.sh + .env + include/)
- Slack notification path on failure

## Diagram (Mermaid)

```mermaid
flowchart TD
    classDef compute fill:#D6EAF8,stroke:#1B4F72,color:#1B2631
    classDef storage fill:#FCF3CF,stroke:#7D6608,color:#1B2631
    classDef external fill:#FADBD8,stroke:#922B21,color:#1B2631
    classDef pipeline fill:#E8DAEF,stroke:#6C3483,color:#1B2631
    classDef task fill:#D5F5E3,stroke:#186A3B,color:#1B2631

    subgraph "Docker Compose Environment"
        direction TB
        Scheduler["Airflow Scheduler"]:::compute
        Webserver["Airflow Webserver"]:::compute
        Workers["Celery Workers"]:::compute
        Redis["Redis Broker"]:::storage
        MetaDB["Postgres Metadata DB"]:::storage
    end

    subgraph "Source & Storage"
        direction TB
        SourceDB["Source PostgreSQL (bookings)"]:::storage
        MinIO["MinIO Bucket (extracted-data)"]:::storage
    end

    subgraph "Staging Pipeline" 
        direction TB
        STAG_DAG["Staging DAG\n(run.py)"]:::pipeline
        Extract["Extract Task"]:::task
        Load["Load Task"]:::task
        TransformStg["Transform Task"]:::task
    end

    subgraph "Warehouse Pipeline"
        direction TB
        WH_DAG["Warehouse DAG\n(run.py)"]:::pipeline
        DBT["DBT Engine"]:::compute
    end

    subgraph "Storage DBs"
        direction TB
        StagingDB["Staging Postgres Schema"]:::storage
        WarehouseDB["Warehouse Postgres DB"]:::storage
    end

    Slack["Slack Webhook"]:::external

    Scheduler -->|"triggers"| STAG_DAG
    STAG_DAG -->|"runs"| Extract
    Extract -->|"reads from"| SourceDB
    Extract -->|"writes CSV"| MinIO
    Extract -->|"pushes metadata (XCom)"| Load
    Load -->|"pulls CSV"| MinIO
    Load -->|"upsert via pangres"| StagingDB
    Load -->|"calls Transform"| TransformStg
    TransformStg -->|"on success TriggerDagRun"| WH_DAG
    WH_DAG -->|"runs DBT models"| DBT
    DBT -->|"writes dimensional/fact tables"| WarehouseDB

    %% Failure notification paths
    Extract -->|"on failure"| Slack
    Load -->|"on failure"| Slack
    TransformStg -->|"on failure"| Slack
    DBT -->|"on failure"| Slack

    %% Configuration & Helpers
    subgraph "Configuration & Helpers"
        direction TB
        StartSh["start.sh"]:::external
        ConnYAML["connections.yaml"]:::external
        VarsJSON["variables.json"]:::external
        Fernet["fernet.py"]:::external
        MinioPy["minio.py"]:::external
        PostgresPy["postgres.py"]:::external
        SlackCB["slack_notifier.py"]:::external
    end

    StartSh --> Scheduler
    StartSh --> Webserver
    ConnYAML --> StartSh
    VarsJSON --> StartSh
    Fernet --> Webserver
    MinioPy --> Extract
    PostgresPy --> Load
    SlackCB --> Extract
    SlackCB --> Load
    SlackCB --> TransformStg
    SlackCB --> DBT

    %% DBT project structure
    subgraph "DBT Project (pacflight_dbt)"
        direction TB
        DBTProj["dbt_project.yml"]:::external
        Seeds["seeds/"]:::external
        Snapshots["snapshots/"]:::external
        Logs["logs/"]:::external
    end
    DBT --> DBTProj
    DBT --> Seeds
    DBT --> Snapshots
    DBT --> Logs

    %% Click Events
    click STAG_DAG "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/dags/flights_staging_pipeline/run.py"
    click WH_DAG "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/dags/flights_warehouse_pipeline/run.py"
    click Extract "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/dags/flights_staging_pipeline/tasks/components/extract.py"
    click Load "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/dags/flights_staging_pipeline/tasks/components/load.py"
    click TransformStg "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/dags/flights_staging_pipeline/tasks/components/transform.py"
    click DBTProj "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/dags/flights_warehouse_pipeline/pacflight_dbt/dbt_project.yml"
    click Seeds "https://github.com/hudiyaresa/airflow-dbt-flightdata/tree/main/dags/flights_warehouse_pipeline/pacflight_dbt/seeds/"
    click Snapshots "https://github.com/hudiyaresa/airflow-dbt-flightdata/tree/main/dags/flights_warehouse_pipeline/pacflight_dbt/snapshots/"
    click Logs "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/dags/flights_warehouse_pipeline/pacflight_dbt/logs/dbt.log"
    click StartSh "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/start.sh"
    click ConnYAML "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/include/connections.yaml"
    click VarsJSON "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/include/variables.json"
    click Fernet "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/fernet.py"
    click MinioPy "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/dags/helper/minio.py"
    click PostgresPy "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/dags/helper/postgres.py"
    click SlackCB "https://github.com/hudiyaresa/airflow-dbt-flightdata/blob/main/dags/helper/callbacks/slack_notifier.py"
```
