How nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow- works
This repository implements an Airbnb streaming data pipeline: scheduled Lambda invocations generate synthetic, intentionally imperfect entity records, publish them to Kinesis, and ultimately feed Snowflake transformations and analytical models. Airflow runs the dbt layers on a six-hour schedule, with tests gating progression. The graph follows the documented AWS ingestion path and dbt stages; infrastructure wiring beyond the excerpts is based on README descriptions, not inspected deployment code.
Main entry points: airbnb_dbt_pipeline.py Airflow DAG · silver Silver models · lambda_handler.py Lambda handler · generator.py Synthetic generator
Components 18
Data generation
- Lambda handler
lambda/lambda_handler.py - Synthetic generator
lambda/generator.py
Streaming ingestion
- Kinesis streams
- Kinesis Firehose
- Partitioned S3
- SQS notifications
- Snowpipe
Warehouse modeling
- Snowflake RAW
- Bronze models
airbnb_de_project_dbt/models/bronze - Silver models
airbnb_de_project_dbt/models/silver - Invalid bookings
airbnb_de_project_dbt/models/silver/quarantine_invalid_bookings.sql - SCD-2 snapshots
airbnb_de_project_dbt/snapshots - Date dimension seed
airbnb_de_project_dbt/seeds/dim_date.csv - Gold dimensions
airbnb_de_project_dbt/models/gold - Gold facts
airbnb_de_project_dbt/models/gold - Bookings OBT
airbnb_de_project_dbt/models/gold/gold_obt_bookings.sql
Orchestration
- Airflow DAG
airflow_dags/airbnb_dbt_pipeline.py
Other components
- EventBridge
Connections 18
- EventBridge → Lambda handler: invokes
README.md - Lambda handler → Synthetic generator: runs batches
lambda/lambda_handler.py - Synthetic generator → Kinesis streams: publishes records
README.md - Kinesis streams → Kinesis Firehose: delivers records
README.md - Kinesis Firehose → Partitioned S3: writes partitions
README.md - Partitioned S3 → SQS notifications: notifies on PUT
README.md - SQS notifications → Snowpipe: triggers loading
README.md - Snowpipe → Snowflake RAW: loads tables
README.md - Airflow DAG → Bronze models: runs dbt
airflow_dags/airbnb_dbt_pipeline.py - Bronze models → Silver models: transforms
README.md - Airflow DAG → Silver models: runs after tests
airflow_dags/airbnb_dbt_pipeline.py - Silver models → Invalid bookings: isolates invalid bookings
- Airflow DAG → SCD-2 snapshots: runs snapshot
airflow_dags/airbnb_dbt_pipeline.py - Silver models → SCD-2 snapshots: provides snapshot inputs
README.md - Airflow DAG → Gold dimensions: runs after snapshot
airflow_dags/airbnb_dbt_pipeline.py - Airflow DAG → Gold facts: runs dbt
airflow_dags/airbnb_dbt_pipeline.py - Airflow DAG → Bookings OBT: runs dbt
airflow_dags/airbnb_dbt_pipeline.py - Date dimension seed → Gold dimensions: supports date dimension