# haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks architecture

> Architecture diagram of the GitHub repository haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks
- Repository: https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks

## Overview

This repository is a dbt e-commerce analytics pipeline for Databricks. Its user-to-result workflow loads CSV seeds into Bronze, transforms data through Silver staging and intermediate models, and builds Gold facts and dimensions; a Type 2 customer snapshot preserves history. The README also documents incremental facts, data-quality tests, exposures, run hooks, and a Databricks job. Source excerpts were not provided, so relationships below rely on README statements; model-level SQL dependencies are not independently verified.

Main entry points: [`models/intermediate`](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/tree/main/models/intermediate) (Intermediate views), [`resources/dbt_shopflow_job.yml`](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/resources/dbt_shopflow_job.yml) (dbt Databricks job), [`models/marts/core/_exposures.yml`](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/models/marts/core/_exposures.yml) (Data exposures), [`seeds`](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/tree/main/seeds) (Raw CSV seeds)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion"]
  node_seeds["Raw CSV seeds"]
  node_bronze[("Bronze data")]
end

subgraph group_silver["Silver transformations"]
  node_staging["Staging views"]
  node_intermediate["Intermediate views"]
end

subgraph group_gold["Gold analytics"]
  node_customer_snapshot["Customer SCD2<br/>[scd_customers.sql]"]
  node_fact_orders["Orders fact<br/>[fct_orders.sql]"]
  node_fact_items["Order items fact"]
  node_product_dim["Product dimension<br/>[dim_products.sql]"]
  node_exposures["Data exposures<br/>[_exposures.yml]"]
end

subgraph group_operations["Runtime and quality"]
  node_quality["Data quality tests"]
  node_hooks["Run hooks<br/>[dbt_project.yml]"]
  node_schema_macro["Schema naming"]
  node_job["dbt Databricks job"]
end

node_analyst(("Analytics user"))
node_databricks[("Databricks")]

node_analyst -->|"runs job"| node_job
node_analyst -->|"loads data"| node_seeds
node_seeds -->|"loads into"| node_bronze
node_bronze -->|"stages raw data"| node_staging
node_staging -.->|"transforms further"| node_intermediate
node_intermediate -.->|"builds marts"| node_fact_orders
node_intermediate -.->|"builds marts"| node_fact_items
node_intermediate -.->|"builds marts"| node_product_dim
node_seeds -->|"snapshots customers"| node_customer_snapshot
node_job -->|"runs dbt"| node_databricks
node_fact_orders -->|"tests results"| node_quality
node_fact_items -->|"tests results"| node_quality
node_customer_snapshot -.->|"documents consumers"| node_exposures
node_fact_orders -->|"documents consumers"| node_exposures
node_fact_items -->|"documents consumers"| node_exposures
node_product_dim -->|"documents consumers"| node_exposures
node_job -->|"runs hooks"| node_hooks
node_job -.->|"uses naming"| node_schema_macro

click node_seeds "https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/tree/main/seeds"
click node_staging "https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/tree/main/models/staging"
click node_intermediate "https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/tree/main/models/intermediate"
click node_customer_snapshot "https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/snapshots/scd_customers.sql"
click node_fact_orders "https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/models/marts/core/fct_orders.sql"
click node_fact_items "https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/models/marts/core/fct_order_items.sql"
click node_product_dim "https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/models/marts/core/dim_products.sql"
click node_exposures "https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/models/marts/core/_exposures.yml"
click node_quality "https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/tree/main/tests"
click node_hooks "https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/dbt_project.yml"
click node_schema_macro "https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/macros/generate_schema_name.sql"
click node_job "https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/resources/dbt_shopflow_job.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_seeds,node_bronze,node_analyst toneBlue
class node_staging,node_intermediate,node_databricks toneAmber
class node_customer_snapshot,node_fact_orders,node_fact_items,node_product_dim,node_exposures toneMint
class node_quality,node_hooks,node_schema_macro,node_job toneRose
```

## Components

### Ingestion

- [Raw CSV seeds](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/tree/main/seeds): `seeds`
- **Bronze data**

### Silver transformations

- [Staging views](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/tree/main/models/staging): `models/staging`
- [Intermediate views](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/tree/main/models/intermediate): `models/intermediate`

### Gold analytics

- [Customer SCD2](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/snapshots/scd_customers.sql): `snapshots/scd_customers.sql`
- [Orders fact](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/models/marts/core/fct_orders.sql): `models/marts/core/fct_orders.sql`
- [Order items fact](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/models/marts/core/fct_order_items.sql): `models/marts/core/fct_order_items.sql`
- [Product dimension](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/models/marts/core/dim_products.sql): `models/marts/core/dim_products.sql`
- [Data exposures](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/models/marts/core/_exposures.yml): `models/marts/core/_exposures.yml`

### Runtime and quality

- [Data quality tests](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/tree/main/tests): `tests`
- [Run hooks](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/dbt_project.yml): `dbt_project.yml`
- [Schema naming](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/macros/generate_schema_name.sql): `macros/generate_schema_name.sql`
- [dbt Databricks job](https://github.com/haiderabbasi6357-cloud/project-end-to-end-dbt-elt-data-warehouse-on-databricks/blob/main/resources/dbt_shopflow_job.yml): `resources/dbt_shopflow_job.yml`

### Other components

- **Analytics user**
- **Databricks**

## Connections

- Analytics user → dbt Databricks job: runs job (evidence: `README.md`)
- Analytics user → Raw CSV seeds: loads data (evidence: `README.md`)
- Raw CSV seeds → Bronze data: loads into (evidence: `README.md`)
- Bronze data → Staging views: stages raw data (evidence: `README.md`)
- Staging views → Intermediate views: transforms further
- Intermediate views → Orders fact: builds marts
- Intermediate views → Order items fact: builds marts
- Intermediate views → Product dimension: builds marts
- Raw CSV seeds → Customer SCD2: snapshots customers (evidence: `README.md`)
- dbt Databricks job → Databricks: runs dbt (evidence: `README.md`)
- Orders fact → Data quality tests: tests results (evidence: `README.md`)
- Order items fact → Data quality tests: tests results (evidence: `README.md`)
- Customer SCD2 → Data exposures: documents consumers
- Orders fact → Data exposures: documents consumers (evidence: `README.md`)
- Order items fact → Data exposures: documents consumers (evidence: `README.md`)
- Product dimension → Data exposures: documents consumers (evidence: `README.md`)
- dbt Databricks job → Run hooks: runs hooks (evidence: `README.md`)
- dbt Databricks job → Schema naming: uses naming
