# nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow- architecture

> Architecture diagram of the GitHub repository nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-
- Repository: https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-
- Video tour (about a minute): https://gitdiagram.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/video

## Overview

This repository implements an Airbnb streaming data pipeline: scheduled Lambda invocations generate synthetic, intentionally imperfect entity records, publish them to Kinesis, and ultimately feed Snowflake transformations and analytical models. Airflow runs the dbt layers on a six-hour schedule, with tests gating progression. The graph follows the documented AWS ingestion path and dbt stages; infrastructure wiring beyond the excerpts is based on README descriptions, not inspected deployment code.

Main entry points: [`airflow_dags/airbnb_dbt_pipeline.py`](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/airflow_dags/airbnb_dbt_pipeline.py) (Airflow DAG), [`airbnb_de_project_dbt/models/silver`](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/tree/main/airbnb_de_project_dbt/models/silver) (Silver models), [`lambda/lambda_handler.py`](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/lambda/lambda_handler.py) (Lambda handler), [`lambda/generator.py`](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/lambda/generator.py) (Synthetic generator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_generation["Data generation"]
  node_lambda["Lambda handler<br/>[lambda_handler.py]"]
  node_generator["Synthetic generator<br/>[generator.py]"]
end

subgraph group_ingestion["Streaming ingestion"]
  node_kinesis["Kinesis streams"]
  node_firehose["Kinesis Firehose"]
  node_s3[("Partitioned S3")]
  node_sqs["SQS notifications"]
  node_snowpipe["Snowpipe"]
end

subgraph group_warehouse["Warehouse modeling"]
  node_raw[("Snowflake RAW")]
  node_bronze[("Bronze models")]
  node_silver[("Silver models")]
  node_quarantine[("Invalid bookings")]
  node_snapshots[("SCD-2 snapshots")]
  node_date_seed[("Date dimension seed<br/>[dim_date.csv]")]
  node_gold_dims[("Gold dimensions")]
  node_gold_facts[("Gold facts")]
  node_gold_obt[("Bookings OBT")]
end

subgraph group_orchestration["Orchestration"]
  node_airflow["Airflow DAG"]
end

node_eventbridge(("EventBridge"))

node_eventbridge -->|"invokes"| node_lambda
node_lambda -->|"runs batches"| node_generator
node_generator -->|"publishes records"| node_kinesis
node_kinesis -->|"delivers records"| node_firehose
node_firehose -->|"writes partitions"| node_s3
node_s3 -->|"notifies on PUT"| node_sqs
node_sqs -->|"triggers loading"| node_snowpipe
node_snowpipe -->|"loads tables"| node_raw
node_airflow -->|"runs dbt"| node_bronze
node_bronze -->|"transforms"| node_silver
node_airflow -->|"runs after tests"| node_silver
node_silver -.->|"isolates invalid bookings"| node_quarantine
node_airflow -->|"runs snapshot"| node_snapshots
node_silver -->|"provides snapshot inputs"| node_snapshots
node_airflow -->|"runs after snapshot"| node_gold_dims
node_airflow -->|"runs dbt"| node_gold_facts
node_airflow -->|"runs dbt"| node_gold_obt
node_date_seed -.->|"supports date dimension"| node_gold_dims

click node_lambda "https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/lambda/lambda_handler.py"
click node_generator "https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/lambda/generator.py"
click node_bronze "https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/tree/main/airbnb_de_project_dbt/models/bronze"
click node_silver "https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/tree/main/airbnb_de_project_dbt/models/silver"
click node_quarantine "https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/airbnb_de_project_dbt/models/silver/quarantine_invalid_bookings.sql"
click node_snapshots "https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/tree/main/airbnb_de_project_dbt/snapshots"
click node_date_seed "https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/airbnb_de_project_dbt/seeds/dim_date.csv"
click node_gold_dims "https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/tree/main/airbnb_de_project_dbt/models/gold"
click node_gold_facts "https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/tree/main/airbnb_de_project_dbt/models/gold"
click node_gold_obt "https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/airbnb_de_project_dbt/models/gold/gold_obt_bookings.sql"
click node_airflow "https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/airflow_dags/airbnb_dbt_pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_lambda,node_generator toneBlue
class node_kinesis,node_firehose,node_s3,node_sqs,node_snowpipe toneAmber
class node_raw,node_bronze,node_silver,node_quarantine,node_snapshots,node_date_seed,node_gold_dims,node_gold_facts,node_gold_obt toneMint
class node_airflow toneRose
class node_eventbridge toneIndigo
```

## Components

### Data generation

- [Lambda handler](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/lambda/lambda_handler.py): `lambda/lambda_handler.py`
- [Synthetic generator](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/lambda/generator.py): `lambda/generator.py`

### Streaming ingestion

- **Kinesis streams**
- **Kinesis Firehose**
- **Partitioned S3**
- **SQS notifications**
- **Snowpipe**

### Warehouse modeling

- **Snowflake RAW**
- [Bronze models](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/tree/main/airbnb_de_project_dbt/models/bronze): `airbnb_de_project_dbt/models/bronze`
- [Silver models](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/tree/main/airbnb_de_project_dbt/models/silver): `airbnb_de_project_dbt/models/silver`
- [Invalid bookings](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/airbnb_de_project_dbt/models/silver/quarantine_invalid_bookings.sql): `airbnb_de_project_dbt/models/silver/quarantine_invalid_bookings.sql`
- [SCD-2 snapshots](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/tree/main/airbnb_de_project_dbt/snapshots): `airbnb_de_project_dbt/snapshots`
- [Date dimension seed](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/airbnb_de_project_dbt/seeds/dim_date.csv): `airbnb_de_project_dbt/seeds/dim_date.csv`
- [Gold dimensions](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/tree/main/airbnb_de_project_dbt/models/gold): `airbnb_de_project_dbt/models/gold`
- [Gold facts](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/tree/main/airbnb_de_project_dbt/models/gold): `airbnb_de_project_dbt/models/gold`
- [Bookings OBT](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/airbnb_de_project_dbt/models/gold/gold_obt_bookings.sql): `airbnb_de_project_dbt/models/gold/gold_obt_bookings.sql`

### Orchestration

- [Airflow DAG](https://github.com/nirmalkumar31/airbnb-de-project-aws-dbt-snowflake-airflow-/blob/main/airflow_dags/airbnb_dbt_pipeline.py): `airflow_dags/airbnb_dbt_pipeline.py`

### Other components

- **EventBridge**

## Connections

- EventBridge → Lambda handler: invokes (evidence: `README.md`)
- Lambda handler → Synthetic generator: runs batches (evidence: `lambda/lambda_handler.py`)
- Synthetic generator → Kinesis streams: publishes records (evidence: `README.md`)
- Kinesis streams → Kinesis Firehose: delivers records (evidence: `README.md`)
- Kinesis Firehose → Partitioned S3: writes partitions (evidence: `README.md`)
- Partitioned S3 → SQS notifications: notifies on PUT (evidence: `README.md`)
- SQS notifications → Snowpipe: triggers loading (evidence: `README.md`)
- Snowpipe → Snowflake RAW: loads tables (evidence: `README.md`)
- Airflow DAG → Bronze models: runs dbt (evidence: `airflow_dags/airbnb_dbt_pipeline.py`)
- Bronze models → Silver models: transforms (evidence: `README.md`)
- Airflow DAG → Silver models: runs after tests (evidence: `airflow_dags/airbnb_dbt_pipeline.py`)
- Silver models → Invalid bookings: isolates invalid bookings
- Airflow DAG → SCD-2 snapshots: runs snapshot (evidence: `airflow_dags/airbnb_dbt_pipeline.py`)
- Silver models → SCD-2 snapshots: provides snapshot inputs (evidence: `README.md`)
- Airflow DAG → Gold dimensions: runs after snapshot (evidence: `airflow_dags/airbnb_dbt_pipeline.py`)
- Airflow DAG → Gold facts: runs dbt (evidence: `airflow_dags/airbnb_dbt_pipeline.py`)
- Airflow DAG → Bookings OBT: runs dbt (evidence: `airflow_dags/airbnb_dbt_pipeline.py`)
- Date dimension seed → Gold dimensions: supports date dimension
