How hudiyaresa/airflow-dbt-flightdata works
Project Type & Purpose
• This is an orchestrated ELT (Extract, Load, Transform) pipeline built on Apache Airflow.
• It extracts flight‐booking data from a PostgreSQL source, lands CSVs in MinIO, loads into a staging Postgres schema, then transforms into dimensional/fact tables via DBT.
• It runs inside Docker Compose, uses CeleryExecutor for scaling, supports Slack alerting, Jinja templating, XComs and incremental loads.
File Structure & Key Artifacts
• Top-level:
• Dockerfile, docker-compose.yml, start.sh
• / (Airflow CLI variables & connections)
• dags/ (two DAGs: flights_staging_pipeline, flights_warehouse_pipeline)
• helper/ (MinIO & Postgres connectors, Slack callback)
• data/ (init SQL for source & warehouse)
• dbt‐requirements.txt, requirements.txt
• flights_staging_pipeline:
• run.py (DAG definition & scheduling)
• tasks/components: extract.py, load.py, transform.py (modular operators)
• flights_warehouse_pipeline:
• run.py triggers DBT (via TriggerDagRunOperator)
• pacflight_dbt/ (dbt_project.yml, snapshots, seeds, logs)
• Config:
• .env (Airflow and service credentials)
• /connections.yaml, include/variables.json loaded via start.sh
Extract Architectural Insights from README
• Orchestrator: Airflow (Scheduler, Webserver, Workers via CeleryExecutor)
• Source DB: PostgreSQL (bookings schema)
• Data Lake: MinIO (“extracted-data” bucket)
• Data Warehouse:
• Staging Schema: Postgres via Upsert (pangres)
• Warehouse Schema: DBT models (dimensional/fact tables)
• Notification: Slack Webhook (failure alerts)
• Deployment: Docker Compose
Main Components & Relationships
a. Components:
• Docker Compose environment
• Airflow Scheduler, Webserver, Celery Workers
• Redis (Broker), Postgres (Metadata DB)
• Source Postgres DB
• MinIO Object Store
• Staging Postgres DB
• DBT engine (inside Airflow task)
• Warehouse Postgres DB
• Slack endpoint
b. Interactions:
1. Scheduler triggers staging DAG
2. Extract task → reads from Source DB → writes CSV to MinIO
3. XCom passes file metadata → Load task pulls from MinIO → Upserts into Staging DB
4. TriggerDagRunOperator → kicks off warehouse DAG
5. Warehouse DAG runs DBT models against Staging → writes to Warehouse DB
6. On failure callbacks → helper.slack_notifier → Slack Webhook
c. Architectural Patterns:
• Modular task components (extract/load/transform)
• Dynamic task creation via Jinja templates
• XCom for lightweight metadata passing
• Upsert pattern for idempotent loads (pangres)
• TriggerDagRun for pipeline chaining
• DBT for transformations (snapshot/seed/model)
• CeleryExecutor for distributed task execution
d. Technologies & Frameworks:
• Python + Airflow
• Docker Compose, Celery, Redis
• PostgreSQL, MinIO
• DBT, Pangres
• Slack Webhooks
Essential Elements
• On environment/config loading (start.sh + .env + include/)
• Slack notification path on failure