# bernard-mlab/dbt-airbnb architecture

> Architecture diagram of the GitHub repository bernard-mlab/dbt-airbnb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bernard-mlab/dbt-airbnb
- Repository: https://github.com/bernard-mlab/dbt-airbnb

## Overview

##### Major Components
- DBT Core Configuration: Represent the central configuration files (e.g., dbt_project.yml, pyproject.toml) that orchestrate the entire project.
- Data Sources & Ingestion:
- Seeds: e.g., seeds/seed_full_moon_dates.csv.
- Source definitions: Found in files like models/sources.yml and within models/src (src_hosts.sql, src_listings.sql, src_reviews.sql).
- Data Transformations:
- Staging / Raw Data: Use snapshots (e.g., snapshots/scd_raw_hosts.sql, snapshots/scd_raw_listings.sql) to capture historical changes.
- Dimensional Models: In the models/dim folder (dim_hosts_cleansed.sql, dim_listings_cleansed.sql, etc.) that perform cleansing and transformation.
- Fact Models: In the models/fct folder (e.g., fct_reviews.sql) to capture the measures.
- Data Marts: In the models/mart folder (e.g., mart_fullmoon_reviews.sql) which aggregate data for analytics.
- Analytical Views & Dashboards:
- Dashboards configuration (models/dashboards.yml) and documentation (models/docs.md, models/overview.md).
- Macros: Custom SQL snippets or functions that encapsulate business logic (see files like macros/no_nulls_in_columns.sql, macros/positive_value.sql).
- Testing: Contains validation SQL tests in the tests directory ensuring data quality–for example, tests/consistent_created_at.sql, tests/no_nulls_in_dim_listings.sql, etc.
- Analyses: Contains exploratory or ad-hoc queries (analyses/full_moon_no_sleep.sql).

##### Relationships and Data Flow
- Data flows from source definitions (models/src) into staging (snapshots) or directly into dimensional models (models/dim).
- Dimensional models then feed into fact tables (models/fct), which in turn contribute to data marts (models/mart) for higher level analytics.
- Interactions with macros: indicate that reusable logic is injected into transformation SQL, bridging various layers.
- Testing components validate each step so indicate they overlay the transformation process, ensuring data quality at every stage.
- Optionally, include an “analyses” block showcasing how analytical SQL queries extract insights from the transformed data.

##### Architectural Patterns and Design Principles
- Layered Architecture: The separation between sources, staging, transformation layers (dimensions, facts, marts), and testing emphasizes modularity.
- Reusability through Macros: The project makes use of macros to enforce consistency and reduce redundancy across SQL models.
- Testing as an Integral Part: Tests are embedded into the workflow, ensuring that each transformation step adheres to quality standards.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% DBT Core Configuration
    subgraph "DBT Core Configuration"
        config1["dbt_project.yml"]:::config
        config2["pyproject.toml"]:::config
    end

    %% Data Ingestion
    subgraph "Data Ingestion"
        subgraph "Seeds"
            seed1["seed_full_moon_dates.csv"]:::ingestion
        end
        subgraph "Source Definitions"
            srcYml["sources.yml"]:::ingestion
            srcHosts["src_hosts.sql"]:::ingestion
            srcListings["src_listings.sql"]:::ingestion
            srcReviews["src_reviews.sql"]:::ingestion
        end
    end

    %% Snapshots
    subgraph "Snapshots"
        snap1["scd_raw_hosts.sql"]:::ingestion
        snap2["scd_raw_listings.sql"]:::ingestion
    end

    %% Dimensional Models
    subgraph "Dimensional Models"
        dim1["dim_hosts_cleansed.sql"]:::transformation
        dim2["dim_listings_cleansed.sql"]:::transformation
        dim3["dim_listings_w_hosts.sql"]:::transformation
    end

    %% Fact Models
    subgraph "Fact Models"
        fact1["fct_reviews.sql"]:::transformation
    end

    %% Data Marts
    subgraph "Data Marts"
        mart1["mart_fullmoon_reviews.sql"]:::transformation
    end

    %% Analytical Views & Dashboards
    subgraph "Analytical Views & Dashboards"
        dash1["dashboards.yml"]:::analytics
        docs1["docs.md"]:::analytics
        overview1["overview.md"]:::analytics
    end

    %% Macros
    subgraph "Macros"
        macro1["no_nulls_in_columns.sql"]:::macro
        macro2["positive_value.sql"]:::macro
    end

    %% Testing
    subgraph "Testing"
        test1["consistent_created_at.sql"]:::testing
        test2["dim_listings_minimum_nights.sql"]:::testing
        test3["no_nulls_in_dim_listings.sql"]:::testing
    end

    %% Analyses
    subgraph "Analyses"
        analysis1["full_moon_no_sleep.sql"]:::analysis
    end

    %% Data Flow Connections
    config1 -->|"configures"| seed1
    config1 -->|"configures"| srcYml
    config2 -->|"configures"| srcHosts

    seed1 -->|"feeds"| snap1
    srcYml -->|"defines"| snap1
    srcHosts -->|"defines"| snap1
    srcListings -->|"defines"| snap2
    srcReviews -->|"defines"| snap2

    snap1 -->|"transforms"| dim1
    snap2 -->|"transforms"| dim2
    snap2 -->|"transforms"| dim3

    dim1 -->|"aggregates"| fact1
    dim2 -->|"aggregates"| fact1
    dim3 -->|"aggregates"| fact1

    fact1 -->|"feeds"| mart1
    mart1 -->|"drives"| dash1
    mart1 -->|"drives"| docs1
    mart1 -->|"drives"| overview1
    mart1 -->|"drives"| analysis1

    %% Macros interact with Transformation Layers
    macro1 -.->|"injects"| dim1
    macro1 -.->|"injects"| fact1
    macro1 -.->|"injects"| mart1
    macro2 -.->|"injects"| dim2
    macro2 -.->|"injects"| fact1
    macro2 -.->|"injects"| mart1

    %% Testing overlay on Transformation Layers
    test1 ---|"validates"| dim1
    test2 ---|"validates"| fact1
    test3 ---|"validates"| mart1

    %% Click Events for DBT Core Configuration
    click config1 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/dbt_project.yml"
    click config2 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/pyproject.toml"

    %% Click Events for Seeds
    click seed1 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/seeds/seed_full_moon_dates.csv"

    %% Click Events for Source Definitions
    click srcYml "https://github.com/bernard-mlab/dbt-airbnb/blob/main/models/sources.yml"
    click srcHosts "https://github.com/bernard-mlab/dbt-airbnb/blob/main/models/src/src_hosts.sql"
    click srcListings "https://github.com/bernard-mlab/dbt-airbnb/blob/main/models/src/src_listings.sql"
    click srcReviews "https://github.com/bernard-mlab/dbt-airbnb/blob/main/models/src/src_reviews.sql"

    %% Click Events for Snapshots
    click snap1 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/snapshots/scd_raw_hosts.sql"
    click snap2 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/snapshots/scd_raw_listings.sql"

    %% Click Events for Dimensional Models
    click dim1 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/models/dim/dim_hosts_cleansed.sql"
    click dim2 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/models/dim/dim_listings_cleansed.sql"
    click dim3 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/models/dim/dim_listings_w_hosts.sql"

    %% Click Events for Fact Models
    click fact1 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/models/fct/fct_reviews.sql"

    %% Click Events for Data Marts
    click mart1 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/models/mart/mart_fullmoon_reviews.sql"

    %% Click Events for Analytical Views & Dashboards
    click dash1 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/models/dashboards.yml"
    click docs1 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/models/docs.md"
    click overview1 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/models/overview.md"

    %% Click Events for Macros
    click macro1 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/macros/no_nulls_in_columns.sql"
    click macro2 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/macros/positive_value.sql"

    %% Click Events for Testing
    click test1 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/tests/consistent_created_at.sql"
    click test2 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/tests/dim_listings_minimum_nights.sql"
    click test3 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/tests/no_nulls_in_dim_listings.sql"

    %% Click Event for Analyses
    click analysis1 "https://github.com/bernard-mlab/dbt-airbnb/blob/main/analyses/full_moon_no_sleep.sql"

    %% Styles
    classDef config fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef ingestion fill:#FFE4B5,stroke:#000,stroke-width:2px;
    classDef transformation fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef analytics fill:#E6E6FA,stroke:#000,stroke-width:2px;
    classDef macro fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef testing fill:#FFB6C1,stroke:#000,stroke-width:2px;
    classDef analysis fill:#D8BFD8,stroke:#000,stroke-width:2px;
```
